Not the answer you need?
Register and ask your own question!

Strange iowait peaks on CPU Usage

Victor SafronovVictor Safronov ContributorInactive User Role Beginner
We noticed strange peaks on CPU Usage. We do not see anything strange in server behavior, we also don't see these peaks in Zabbix and sysstat daily output.
When I set interval to 1 sec, it shows iowait usage at level 20000-30000%. Can it be a bug somewhere in node_exporter or pmm itself?

pmm-client-1.1.1-1.x86_64

Comments

  • MykolaMykola Percona Percona Staff Role
    Hi Victor,
    we also don't see these peaks in Zabbix and sysstat daily output.
    it is expected because PMM very accurate, it reads statuses one per second.
    usual monitoring tools (like zabbix and sysstat) show average value for a minute and it is hard to troubleshoot database load based on such average values.
    it shows iowait usage at level 20000-30000%
    
    as you know, pmm-client use prometheus/node_exporter for collecting cpu usage.
    I inspected node_exporter code
    https://github.com/prometheus/node_exporter/blob/master/collector/stat_linux.go#L101-L121
    looks like it just reads /proc/stat file and pass kernel counter values "as is" to prometheus, without any additional processing.
    so it is cannot be node_exporter error.

    can you share screenshot with cpu graph from prometheus?
    http://pmm-server-ip/prometheus/graph?g0.range_input=1h&g0.expr=node_cpu%7Binstance%3D%22some-server%22%7D&g0.tab=0
    
    please replace "some-server" by problematic server hostname (it is visible in grafan dropdown list)
  • Victor SafronovVictor Safronov Contributor Inactive User Role Beginner
    Well, this graph looks quite strange and useless :) I've expanded the time period to 6 hours, but it's the same as for 1h.
  • MykolaMykola Percona Percona Staff Role
    1. do you use virtualization with shared cpu core usage? (when one core sometimes shared between several VMs)
    2. can you try to use the following Query in graph?
    sum(rate(node_cpu{instance="$host", mode!="idle"}[$interval])) by (mode)* 100 / scalar(sum(rate(node_cpu{instance="$host"}[$interval]))) or sum(irate(node_cpu{instance="$host", mode!="idle"}[5m])) by (mode) * 100 / scalar(sum(irate(node_cpu{instance="$host"}[5m])))
    
    it is needed to click on graph name, open for edit, open "metrics" tab and replace Query.
    please find screenshot
  • Victor SafronovVictor Safronov Contributor Inactive User Role Beginner
    Mykola wrote: »
    1. do you use virtualization with shared cpu core usage? (when one core sometimes shared between several VMs)
    2. can you try to use the following Query in graph?
    1. No, pmm-client is set up on the real server.
    2. Is it a new feature? I see only 'View' and 'Share' links, and no 'Edit'. I work with admin account
  • MykolaMykola Percona Percona Staff Role
    what version of PMM Server do you use?
    can you click on Grafana logo and check user name?
  • Victor SafronovVictor Safronov Contributor Inactive User Role Beginner
    Mykola wrote: »
    what version of PMM Server do you use?
    can you click on Grafana logo and check user name?

    We just upgraded to 1.1.2 and nothing changed. Is there a hardcoded check for username or user id? We accidentally deleted the default admin account just after installing and creating custom accounts with admin rights.
  • MykolaMykola Percona Percona Staff Role
    did you set SERVER_PASSWORD variable during container run?
  • Victor SafronovVictor Safronov Contributor Inactive User Role Beginner
    Mykola wrote: »
    did you set SERVER_PASSWORD variable during container run?

    No, these variables are not mentioned here, so we didn't know about it. We should use it to access Edit tab?
  • MykolaMykola Percona Percona Staff Role
    No, it is not needed to use SERVER_PASSWORD (but recommended, see https://www.percona.com/doc/percona-monitoring-and-management/security.html for more details)
    i just wondered about authentication settings.

    in 1.1.2 without SERVER_PASSWORD option - any unauthorised user should have permissions to modify graphs, so can you try to press "sign out" button and modify graphs?
  • Victor SafronovVictor Safronov Contributor Inactive User Role Beginner
    After signing out it doesn't allow me to do anything but seeing /graph/login page
  • MykolaMykola Percona Percona Staff Role
    interesting... can you try to open "Incognito Window" in you browser and try to open http://PMM-Server-IP/graph/ url? it should work without auth under guest user.

    can you share output of the following command?
    docker inspect pmm-server
    
  • Victor SafronovVictor Safronov Contributor Inactive User Role Beginner
    Mykola wrote: »
    interesting... can you try to open "Incognito Window" in you browser and try to open http://PMM-Server-IP/graph/ url? it should work without auth under guest user.

    can you share output of the following command?
    docker inspect pmm-server
    
    No, I'm still redirected to /graph/login
    [
        {
            "Id": "7ea43a1490b83e332f7313ecf090f24333dfee9e2924d9302dbabe4e34480f29",
            "Created": "2017-04-19T11:45:00.618581204Z",
            "Path": "/opt/entrypoint.sh",
            "Args": [],
            "State": {
                "Status": "running",
                "Running": true,
                "Paused": false,
                "Restarting": false,
                "OOMKilled": false,
                "Dead": false,
                "Pid": 9526,
                "ExitCode": 0,
                "Error": "",
                "StartedAt": "2017-04-19T11:45:06.156467447Z",
                "FinishedAt": "0001-01-01T00:00:00Z"
            },
            "Image": "sha256:38b9f4754f773c192d0e72ea65a400ee3b42e34b4a9a9505e278bdc0453b31c6",
            "ResolvConfPath": "/var/lib/docker/containers/7ea43a1490b83e332f7313ecf090f24333dfee9e2924d9302dbabe4e34480f29/resolv.conf",
            "HostnamePath": "/var/lib/docker/containers/7ea43a1490b83e332f7313ecf090f24333dfee9e2924d9302dbabe4e34480f29/hostname",
            "HostsPath": "/var/lib/docker/containers/7ea43a1490b83e332f7313ecf090f24333dfee9e2924d9302dbabe4e34480f29/hosts",
            "LogPath": "",
            "Name": "/pmm-server",
            "RestartCount": 0,
            "Driver": "devicemapper",
            "MountLabel": "",
            "ProcessLabel": "",
            "AppArmorProfile": "",
            "ExecIDs": null,
            "HostConfig": {
                "Binds": null,
                "ContainerIDFile": "",
                "LogConfig": {
                    "Type": "journald",
                    "Config": {}
                },
                "NetworkMode": "default",
                "PortBindings": {
                    "80/tcp": [
                        {
                            "HostIp": "",
                            "HostPort": "81"
                        }
                    ]
                },
                "RestartPolicy": {
                    "Name": "always",
                    "MaximumRetryCount": 0
                },
                "AutoRemove": false,
                "VolumeDriver": "",
                "VolumesFrom": [
                    "pmm-data"
                ],
                "CapAdd": null,
                "CapDrop": null,
                "Dns": [],
                "DnsOptions": [],
                "DnsSearch": [],
                "ExtraHosts": null,
                "GroupAdd": null,
                "IpcMode": "",
                "Cgroup": "",
                "Links": null,
                "OomScoreAdj": 0,
                "PidMode": "",
                "Privileged": false,
                "PublishAllPorts": false,
                "ReadonlyRootfs": false,
                "SecurityOpt": null,
                "UTSMode": "",
                "UsernsMode": "",
                "ShmSize": 67108864,
                "Runtime": "docker-runc",
                "ConsoleSize": [
                    0,
                    0
                ],
                "Isolation": "",
                "CpuShares": 0,
                "Memory": 0,
                "CgroupParent": "",
                "BlkioWeight": 0,
                "BlkioWeightDevice": null,
                "BlkioDeviceReadBps": null,
                "BlkioDeviceWriteBps": null,
                "BlkioDeviceReadIOps": null,
                "BlkioDeviceWriteIOps": null,
                "CpuPeriod": 0,
                "CpuQuota": 0,
                "CpusetCpus": "",
                "CpusetMems": "",
                "Devices": [],
                "DiskQuota": 0,
                "KernelMemory": 0,
                "MemoryReservation": 0,
                "MemorySwap": 0,
                "MemorySwappiness": -1,
                "OomKillDisable": false,
                "PidsLimit": 0,
                "Ulimits": null,
                "CpuCount": 0,
                "CpuPercent": 0,
                "IOMaximumIOps": 0,
                "IOMaximumBandwidth": 0
            },
            "GraphDriver": {
                "Name": "devicemapper",
                "Data": {
                    "DeviceId": "103",
                    "DeviceName": "docker-253:1-3222577957-9b9a13bb42ea97578920df36577feea484dc7247056b586ec055a9490c0dae8a",
                    "DeviceSize": "10737418240"
                }
            },
            "Mounts": [
                {
                    "Name": "3ade038e4a2d102bca1d525c7adced6f3529581882feb16b909d44c0c185c759",
                    "Source": "/var/lib/docker/volumes/3ade038e4a2d102bca1d525c7adced6f3529581882feb16b909d44c0c185c759/_data",
                    "Destination": "/opt/consul-data",
                    "Driver": "local",
                    "Mode": "",
                    "RW": true,
                    "Propagation": ""
                },
                {
                    "Name": "a7d74f024ed1f72f555620ccf3f9fb133685c00a42ab662fb0c4ec7b7b5aa05b",
                    "Source": "/var/lib/docker/volumes/a7d74f024ed1f72f555620ccf3f9fb133685c00a42ab662fb0c4ec7b7b5aa05b/_data",
                    "Destination": "/opt/prometheus/data",
                    "Driver": "local",
                    "Mode": "",
                    "RW": true,
                    "Propagation": ""
                },
                {
                    "Name": "cfaa7da6208461687c48fc1877aaba076d6143345b3e612d334bd6c91f774a51",
                    "Source": "/var/lib/docker/volumes/cfaa7da6208461687c48fc1877aaba076d6143345b3e612d334bd6c91f774a51/_data",
                    "Destination": "/var/lib/grafana",
                    "Driver": "local",
                    "Mode": "",
                    "RW": true,
                    "Propagation": ""
                },
                {
                    "Name": "41a2981c4be53d214c3d0d094cd31a47f1f2e635b5f68069062fb44b3c562c31",
                    "Source": "/var/lib/docker/volumes/41a2981c4be53d214c3d0d094cd31a47f1f2e635b5f68069062fb44b3c562c31/_data",
                    "Destination": "/var/lib/mysql",
                    "Driver": "local",
                    "Mode": "",
                    "RW": true,
                    "Propagation": ""
                }
            ],
            "Config": {
                "Hostname": "7ea43a1490b8",
                "Domainname": "",
                "User": "",
                "AttachStdin": false,
                "AttachStdout": false,
                "AttachStderr": false,
                "ExposedPorts": {
                    "443/tcp": {},
                    "80/tcp": {}
                },
                "Tty": false,
                "OpenStdin": false,
                "StdinOnce": false,
                "Env": [
                    "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
                ],
                "Cmd": [
                    "/opt/entrypoint.sh"
                ],
                "Image": "percona/pmm-server:1.1.2",
                "Volumes": null,
                "WorkingDir": "/opt",
                "Entrypoint": null,
                "OnBuild": null,
                "Labels": {
                    "build-date": "20161214",
                    "license": "GPLv2",
                    "name": "CentOS Base Image",
                    "vendor": "CentOS"
                }
            },
            "NetworkSettings": {
                "Bridge": "",
                "SandboxID": "b8b8dac97f23ffcb59d6cf62ce2d5af8f90bc724ff20dd652d022ba445745e21",
                "HairpinMode": false,
                "LinkLocalIPv6Address": "",
                "LinkLocalIPv6PrefixLen": 0,
                "Ports": {
                    "443/tcp": null,
                    "80/tcp": [
                        {
                            "HostIp": "0.0.0.0",
                            "HostPort": "81"
                        }
                    ]
                },
                "SandboxKey": "/var/run/docker/netns/b8b8dac97f23",
                "SecondaryIPAddresses": null,
                "SecondaryIPv6Addresses": null,
                "EndpointID": "3d2968d423f39e0b77af96e326483578662467ca656c87c6bc629b29b2f757c8",
                "Gateway": "172.17.0.1",
                "GlobalIPv6Address": "",
                "GlobalIPv6PrefixLen": 0,
                "IPAddress": "172.17.0.2",
                "IPPrefixLen": 16,
                "IPv6Gateway": "",
                "MacAddress": "02:42:ac:11:00:02",
                "Networks": {
                    "bridge": {
                        "IPAMConfig": null,
                        "Links": null,
                        "Aliases": null,
                        "NetworkID": "e3bf926620d6b36c55bd4828888a61b69968b2bb0b0f7a43560e5944ddf1d5d3",
                        "EndpointID": "3d2968d423f39e0b77af96e326483578662467ca656c87c6bc629b29b2f757c8",
                        "Gateway": "172.17.0.1",
                        "IPAddress": "172.17.0.2",
                        "IPPrefixLen": 16,
                        "IPv6Gateway": "",
                        "GlobalIPv6Address": "",
                        "GlobalIPv6PrefixLen": 0,
                        "MacAddress": "02:42:ac:11:00:02"
                    }
                }
            }
        }
    ]
    
    
  • MykolaMykola Percona Percona Staff Role
    lets try to add admin permission for your user

    you can open connection to grafana database with this command
    docker exec -it pmm-server sqlite3 /var/lib/grafana/grafana.db
    

    you can list users with following commands
    .mode column
    .header on
    SELECT * FROM user;
    

    add admin permissions for your user with the following command
    UPDATE user SET is_admin=1 WHERE login='your-user';
    
  • Victor SafronovVictor Safronov Contributor Inactive User Role Beginner
    Mykola wrote: »
    UPDATE user SET is_admin=1 WHERE login='your-user';
    

    Well, both our users already have this flag is_admin = 1 in database. Is it a mistery?
    UPD:
    UPDATE user SET id = 1 WHERE id = 2
    
    helped me. Now I see Edit tab
Sign In or Register to comment.

MySQL, InnoDB, MariaDB and MongoDB are trademarks of their respective owners.
Copyright ©2005 - 2020 Percona LLC. All rights reserved.