Strange iowait peaks on CPU Usage

Percona Monitoring and Management (PMM) PMM 1.x
1

We noticed strange peaks on CPU Usage. We do not see anything strange in server behavior, we also don’t see these peaks in Zabbix and sysstat daily output.
When I set interval to 1 sec, it shows iowait usage at level 20000-30000%. Can it be a bug somewhere in node_exporter or pmm itself?

pmm-client-1.1.1-1.x86_64

photoid=48075
photoid=48075764×320 27.5 KB

2

Hi Victor,

it is expected because PMM very accurate, it reads statuses one per second.
usual monitoring tools (like zabbix and sysstat) show average value for a minute and it is hard to troubleshoot database load based on such average values.

it shows iowait usage at level 20000-30000%

as you know, pmm-client use prometheus/node_exporter for collecting cpu usage.
I inspected node_exporter code
https://github.com/prometheus/node_exporter/blob/master/collector/stat_linux.go#L101-L121
looks like it just reads /proc/stat file and pass kernel counter values “as is” to prometheus, without any additional processing.
so it is cannot be node_exporter error.

can you share screenshot with cpu graph from prometheus?

http://pmm-server-ip/prometheus/graph?g0.range_input=1h&g0.expr=node_cpu%7Binstance%3D%22some-server%22%7D&g0.tab=0

please replace “some-server” by problematic server hostname (it is visible in grafan dropdown list)

3

Well, this graph looks quite strange and useless :slight_smile: I’ve expanded the time period to 6 hours, but it’s the same as for 1h.

photoid=48120
photoid=481201371×496 13.1 KB

4
  1. do you use virtualization with shared cpu core usage? (when one core sometimes shared between several VMs)
  2. can you try to use the following Query in graph?
sum(rate(node_cpu{instance="$host", mode!="idle"}[$interval])) by (mode)* 100 / scalar(sum(rate(node_cpu{instance="$host"}[$interval]))) or sum(irate(node_cpu{instance="$host", mode!="idle"}[5m])) by (mode) * 100 / scalar(sum(irate(node_cpu{instance="$host"}[5m])))

it is needed to click on graph name, open for edit, open “metrics” tab and replace Query.
please find screenshot

query.png
query.png1799×1136 46.7 KB

5
  1. No, pmm-client is set up on the real server.
  2. Is it a new feature? I see only ‘View’ and ‘Share’ links, and no ‘Edit’. I work with admin account

photoid=48198
photoid=48198724×532 40.1 KB

6

what version of PMM Server do you use?
can you click on Grafana logo and check user name?

7

We just upgraded to 1.1.2 and nothing changed. Is there a hardcoded check for username or user id? We accidentally deleted the default admin account just after installing and creating custom accounts with admin rights.

8

did you set SERVER_PASSWORD variable during container run?

9

No, these variables are not mentioned here, so we didn’t know about it. We should use it to access Edit tab?

10

No, it is not needed to use SERVER_PASSWORD (but recommended, see [url]Percona Monitoring and Management for more details)
i just wondered about authentication settings.

in 1.1.2 without SERVER_PASSWORD option - any unauthorised user should have permissions to modify graphs, so can you try to press “sign out” button and modify graphs?

11

After signing out it doesn’t allow me to do anything but seeing /graph/login page

12

interesting… can you try to open “Incognito Window” in you browser and try to open [url]http://PMM-Server-IP/graph/[/url] url? it should work without auth under guest user.

can you share output of the following command?

docker inspect pmm-server
13

No, I’m still redirected to /graph/login

[
{
"Id": "7ea43a1490b83e332f7313ecf090f24333dfee9e2924d9302dbabe4e34480f29",
"Created": "2017-04-19T11:45:00.618581204Z",
"Path": "/opt/entrypoint.sh",
"Args": [],
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 9526,
"ExitCode": 0,
"Error": "",
"StartedAt": "2017-04-19T11:45:06.156467447Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Image": "sha256:38b9f4754f773c192d0e72ea65a400ee3b42e34b4a9a9505e278bdc0453b31c6",
"ResolvConfPath": "/var/lib/docker/containers/7ea43a1490b83e332f7313ecf090f24333dfee9e2924d9302dbabe4e34480f29/resolv.conf",
"HostnamePath": "/var/lib/docker/containers/7ea43a1490b83e332f7313ecf090f24333dfee9e2924d9302dbabe4e34480f29/hostname",
"HostsPath": "/var/lib/docker/containers/7ea43a1490b83e332f7313ecf090f24333dfee9e2924d9302dbabe4e34480f29/hosts",
"LogPath": "",
"Name": "/pmm-server",
"RestartCount": 0,
"Driver": "devicemapper",
"MountLabel": "",
"ProcessLabel": "",
"AppArmorProfile": "",
"ExecIDs": null,
"HostConfig": {
"Binds": null,
"ContainerIDFile": "",
"LogConfig": {
"Type": "journald",
"Config": {}
},
"NetworkMode": "default",
"PortBindings": {
"80/tcp": [
{
"HostIp": "",
"HostPort": "81"
}
]
},
"RestartPolicy": {
"Name": "always",
"MaximumRetryCount": 0
},
"AutoRemove": false,
"VolumeDriver": "",
"VolumesFrom": [
"pmm-data"
],
"CapAdd": null,
"CapDrop": null,
"Dns": [],
"DnsOptions": [],
"DnsSearch": [],
"ExtraHosts": null,
"GroupAdd": null,
"IpcMode": "",
"Cgroup": "",
"Links": null,
"OomScoreAdj": 0,
"PidMode": "",
"Privileged": false,
"PublishAllPorts": false,
"ReadonlyRootfs": false,
"SecurityOpt": null,
"UTSMode": "",
"UsernsMode": "",
"ShmSize": 67108864,
"Runtime": "docker-runc",
"ConsoleSize": [
0,
0
],
"Isolation": "",
"CpuShares": 0,
"Memory": 0,
"CgroupParent": "",
"BlkioWeight": 0,
"BlkioWeightDevice": null,
"BlkioDeviceReadBps": null,
"BlkioDeviceWriteBps": null,
"BlkioDeviceReadIOps": null,
"BlkioDeviceWriteIOps": null,
"CpuPeriod": 0,
"CpuQuota": 0,
"CpusetCpus": "",
"CpusetMems": "",
"Devices": [],
"DiskQuota": 0,
"KernelMemory": 0,
"MemoryReservation": 0,
"MemorySwap": 0,
"MemorySwappiness": -1,
"OomKillDisable": false,
"PidsLimit": 0,
"Ulimits": null,
"CpuCount": 0,
"CpuPercent": 0,
"IOMaximumIOps": 0,
"IOMaximumBandwidth": 0
},
"GraphDriver": {
"Name": "devicemapper",
"Data": {
"DeviceId": "103",
"DeviceName": "docker-253:1-3222577957-9b9a13bb42ea97578920df36577feea484dc7247056b586ec055a9490c0dae8a",
"DeviceSize": "10737418240"
}
},
"Mounts": [
{
"Name": "3ade038e4a2d102bca1d525c7adced6f3529581882feb16b909d44c0c185c759",
"Source": "/var/lib/docker/volumes/3ade038e4a2d102bca1d525c7adced6f3529581882feb16b909d44c0c185c759/_data",
"Destination": "/opt/consul-data",
"Driver": "local",
"Mode": "",
"RW": true,
"Propagation": ""
},
{
"Name": "a7d74f024ed1f72f555620ccf3f9fb133685c00a42ab662fb0c4ec7b7b5aa05b",
"Source": "/var/lib/docker/volumes/a7d74f024ed1f72f555620ccf3f9fb133685c00a42ab662fb0c4ec7b7b5aa05b/_data",
"Destination": "/opt/prometheus/data",
"Driver": "local",
"Mode": "",
"RW": true,
"Propagation": ""
},
{
"Name": "cfaa7da6208461687c48fc1877aaba076d6143345b3e612d334bd6c91f774a51",
"Source": "/var/lib/docker/volumes/cfaa7da6208461687c48fc1877aaba076d6143345b3e612d334bd6c91f774a51/_data",
"Destination": "/var/lib/grafana",
"Driver": "local",
"Mode": "",
"RW": true,
"Propagation": ""
},
{
"Name": "41a2981c4be53d214c3d0d094cd31a47f1f2e635b5f68069062fb44b3c562c31",
"Source": "/var/lib/docker/volumes/41a2981c4be53d214c3d0d094cd31a47f1f2e635b5f68069062fb44b3c562c31/_data",
"Destination": "/var/lib/mysql",
"Driver": "local",
"Mode": "",
"RW": true,
"Propagation": ""
}
],
"Config": {
"Hostname": "7ea43a1490b8",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"ExposedPorts": {
"443/tcp": {},
"80/tcp": {}
},
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
],
"Cmd": [
"/opt/entrypoint.sh"
],
"Image": "percona/pmm-server:1.1.2",
"Volumes": null,
"WorkingDir": "/opt",
"Entrypoint": null,
"OnBuild": null,
"Labels": {
"build-date": "20161214",
"license": "GPLv2",
"name": "CentOS Base Image",
"vendor": "CentOS"
}
},
"NetworkSettings": {
"Bridge": "",
"SandboxID": "b8b8dac97f23ffcb59d6cf62ce2d5af8f90bc724ff20dd652d022ba445745e21",
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"Ports": {
"443/tcp": null,
"80/tcp": [
{
"HostIp": "0.0.0.0",
"HostPort": "81"
}
]
},
"SandboxKey": "/var/run/docker/netns/b8b8dac97f23",
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null,
"EndpointID": "3d2968d423f39e0b77af96e326483578662467ca656c87c6bc629b29b2f757c8",
"Gateway": "172.17.0.1",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "172.17.0.2",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"MacAddress": "02:42:ac:11:00:02",
"Networks": {
"bridge": {
"IPAMConfig": null,
"Links": null,
"Aliases": null,
"NetworkID": "e3bf926620d6b36c55bd4828888a61b69968b2bb0b0f7a43560e5944ddf1d5d3",
"EndpointID": "3d2968d423f39e0b77af96e326483578662467ca656c87c6bc629b29b2f757c8",
"Gateway": "172.17.0.1",
"IPAddress": "172.17.0.2",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "02:42:ac:11:00:02"
}
}
}
}
]
14

lets try to add admin permission for your user

you can open connection to grafana database with this command

docker exec -it pmm-server sqlite3 /var/lib/grafana/grafana.db

you can list users with following commands

.mode column
.header on
SELECT * FROM user;

add admin permissions for your user with the following command

UPDATE user SET is_admin=1 WHERE login='your-user';
15

Well, both our users already have this flag is_admin = 1 in database. Is it a mistery?
UPD:

UPDATE user SET id = 1 WHERE id = 2

helped me. Now I see Edit tab