mysqld_exporter use cpu too much

Percona Monitoring and Management (PMM) PMM 1.x
1

Host operating system: output of uname -a
Linux malldbxx 2.6.32-696.3.2.el6.x86_64 #1 SMP Tue Jun 20 01:26:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux mysqld_exporter version: output of mysqld_exporter --version
mysqld_exporter, version 1.4.1 (branch: master, revision: c5b2f15)
build user:
build date:
go version: go1.8 MySQL server version
oracle mysql 5.7.16-log mysqld_exporter command line flags mysql:queries xx.abcd.com - YES pmm:[URL=“unix (witt) · GitHub”]@unix/URL query_source=slowlog, query_examples=false
linux:metrics xx.abcd.com 42000 YES -
mysql:metrics xx.abcd.com 42002 YES pmm:[URL=“unix (witt) · GitHub”]@unix/URL processlist=OFF, tablestats=OFF, userstats=OFF
What did you do that produced an error?
there is no error, it just consume too much cpu resource.
What did you expect to see?
there is no error
What did you see instead?
there is no error.

I just want to know why this deamon use cpu resource too much.

this is “TOP” results in Linux.
as you can see, mysqld_exporter is using 11.7% CPU resource.

thanks.

– “TOP” results

Tasks: 623 total, 2 running, 621 sleeping, 0 stopped, 0 zombie
Cpu(s): 3.7%us, 0.7%sy, 0.0%ni, 95.0%id, 0.4%wa, 0.0%hi, 0.1%si, 0.0%st
Mem: 65842428k total, 62781936k used, 3060492k free, 595428k buffers
Swap: 8191996k total, 620840k used, 7571156k free, 27095728k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
20264 mysql 20 0 42.0g 31g 9232 S 283.6 50.6 284106:34 mysqld
15821 root 20 0 101m 1196 808 R 97.8 0.0 0:01.00 zip
31437 root 20 0 2765m 15m 4220 S 11.7 0.0 0:51.92 mysqld_exporter
2690 root 20 0 0 0 0 S 2.0 0.0 652:54.67 jbd2/sdb1-8
15823 root 20 0 17468 1556 832 R 2.0 0.0 0:00.01 top
1 root 20 0 21396 1292 1076 S 0.0 0.0 0:16.12 init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.21 kthreadd
3 root RT 0 0 0 0 S 0.0 0.0 64:51.99 migration/0
4 root 20 0 0 0 0 S 0.0 0.0 1:52.61 ksoftirqd/0
5 root RT 0 0 0 0 S 0.0 0.0 0:00.00 stopper/0
6 root RT 0 0 0 0 S 0.0 0.0 1:21.87 watchdog/0
7 root RT 0 0 0 0 S 0.0 0.0 75:51.53 migration/1
8 root RT 0 0 0 0 S 0.0 0.0 0:00.00 stopper/1
9 root 20 0 0 0 0 S 0.0 0.0 0:49.07 ksoftirqd/1
10 root RT 0 0 0 0 S 0.0 0.0 1:09.22 watchdog/1
11 root RT 0 0 0 0 S 0.0 0.0 99:10.97 migration/2
12 root RT 0 0 0 0 S 0.0 0.0 0:00.00 stopper/2
13 root 20 0 0 0 0 S 0.0 0.0 0:37.20 ksoftirqd/2
14 root RT 0 0 0 0 S 0.0 0.0 0:44.47 watchdog/2
15 root RT 0 0 0 0 S 0.0 0.0 77:44.68 migration/3
16 root RT 0 0 0 0 S 0.0 0.0 0:00.00 stopper/3
17 root 20 0 0 0 0 S 0.0 0.0 0:31.30 ksoftirqd/3
18 root RT 0 0 0 0 S 0.0 0.0 0:45.33 watchdog/3
19 root RT 0 0 0 0 S 0.0 0.0 46:39.83 migration/4
20 root RT 0 0 0 0 S 0.0 0.0 0:00.00 stopper/4
21 root 20 0 0 0 0 S 0.0 0.0 0:27.29 ksoftirqd/4
22 root RT 0 0 0 0 S 0.0 0.0 0:44.57 watchdog/4
23 root RT 0 0 0 0 S 0.0 0.0 40:09.09 migration/5
24 root RT 0 0 0 0 S 0.0 0.0 0:00.00 stopper/5
25 root 20 0 0 0 0 S 0.0 0.0 0:23.82 ksoftirqd/5
26 root RT 0 0 0 0 S 0.0 0.0 0:39.15 watchdog/5
27 root RT 0 0 0 0 S 0.0 0.0 64:09.10 migration/6
28 root RT 0 0 0 0 S 0.0 0.0 0:00.00 stopper/6
29 root 20 0 0 0 0 S 0.0 0.0 0:21.45 ksoftirqd/6
30 root RT 0 0 0 0 S 0.0 0.0 0:39.43 watchdog/6
31 root RT 0 0 0 0 S 0.0 0.0 62:35.76 migration/7
32 root RT 0 0 0 0 S 0.0 0.0 0:00.00 stopper/7
33 root 20 0 0 0 0 S 0.0 0.0 0:20.04 ksoftirqd/7
34 root RT 0 0 0 0 S 0.0 0.0 0:39.19 watchdog/7
35 root RT 0 0 0 0 S 0.0 0.0 135:55.34 migration/8
36 root RT 0 0 0 0 S 0.0 0.0 0:00.00 stopper/8
37 root 20 0 0 0 0 S 0.0 0.0 3:20.93 ksoftirqd/8
38 root RT 0 0 0 0 S 0.0 0.0 3:21.69 watchdog/8

this is “/var/log/pmm-mysql-metrics.log”

– /var/log/pmm-mysql-metrics.log

time=“2017-11-22T22:12:36+09:00” level=info msg=“Starting mysqld_exporter (version=1.4.1, branch=master, revision=c5b2f15a2b2b46eb53192c6aded039c90f406733)” source=“mysqld_exporter.go:798”
time=“2017-11-22T22:12:36+09:00” level=info msg=“Build context (go=go1.8, user=, date=)” source=“mysqld_exporter.go:799”
time=“2017-11-22T22:12:36+09:00” level=info msg=“HTTPS/TLS is enabled” source=“mysqld_exporter.go:843”
time=“2017-11-22T22:12:36+09:00” level=info msg=“Listening on 106.xxx.xx.xxx:42002” source=“mysqld_exporter.go:846”
time=“2017-11-23T09:42:14+09:00” level=info msg=“Starting mysqld_exporter (version=1.4.1, branch=master, revision=c5b2f15a2b2b46eb53192c6aded039c90f406733)” source=“mysqld_exporter.go:798”
time=“2017-11-23T09:42:14+09:00” level=info msg=“Build context (go=go1.8, user=, date=)” source=“mysqld_exporter.go:799”
time=“2017-11-23T09:42:14+09:00” level=info msg=“HTTPS/TLS is enabled” source=“mysqld_exporter.go:843”
time=“2017-11-23T09:42:14+09:00” level=info msg=“Listening on 106.xxx.xx.xxx:42002” source=“mysqld_exporter.go:846”
time=“2017-11-23T09:42:36+09:00” level=info msg=“Starting mysqld_exporter (version=1.4.1, branch=master, revision=c5b2f15a2b2b46eb53192c6aded039c90f406733)” source=“mysqld_exporter.go:798”
time=“2017-11-23T09:42:36+09:00” level=info msg=“Build context (go=go1.8, user=, date=)” source=“mysqld_exporter.go:799”
time=“2017-11-23T09:42:36+09:00” level=info msg=“HTTPS/TLS is enabled” source=“mysqld_exporter.go:843”
time=“2017-11-23T09:42:36+09:00” level=info msg=“Listening on 106.xxx.xx.xxx:42002” source=“mysqld_exporter.go:846”

2

mysqld_exporter uses a lot of CPU because detailed monitoring does not come free. In fact Majority of MySQL Servers in production tend to have spare CPU capacity so it does not tend to come with any performance impact. If it causes issues for you you can:

  1. Reduce metrics resolution by using -e METRICS_RESOLUTION container option as described here
    [url]Percona Monitoring and Management

  2. Reduce amount of data collected by disabling some of collectors in exporter. This can be done by adding custom parameters to exporters such as
    pmm-admin add mysql:metrics – --collect.perf_schema.eventsstatements=false

3

thank you for your information.

I can reduce the cpu resource a little using this option.

thanks.

4

I am seeing the same pattern.

Three identically spec’ed Dell PowerEdge R720 with Ubuntu 14.04 (trusty) running in a Percona cluster.

Two of the nodes are running pmm-client version 1.1.5-1.trusty where the cpu load for the node_exporter and mysql_exporter is around 3% and one of these is receiving all write load.

One node running is running pmm-client 1.5.0-1.trusty and is consistently using 10-13% cpu for mysqld_exporter and node_exporter.

I have created the following issue:
[URL]https://jira.percona.com/projects/PMM/issues/PMM-1526[/URL]