Not the answer you need?
Register and ask your own question!

PMM is losing instances

2»

Comments

  • StaterosStateros Contributor Current User Role Patron
    Probably no but I don't want change anything while it works fine )))
    CPU usage is high permanently (50-70%). I check prod umm which still on 1.1.1 version. It works on m4.xlarge, support near 50 instances and CPU usage less then on 1.2.0 pmm, near 40-50%.
    For now pmm works with 21 db instances.
  • MykolaMykola Percona Percona Staff Role
    Stateros
    hm, strange, memory using should be bigger, cpu load lower for 21 instance
    should be can you share output of the following command?
    docker inspect pmm-server
    
  • StaterosStateros Contributor Current User Role Patron
    Sure, here
    ~# docker inspect pmm-server
    [
        {
            "Id": "8151d099cf06795b1e2ffa0ed8f23b3b769e70e8d448d507c2f8962afa06b20a",
            "Created": "2017-07-21T06:51:24.176130151Z",
            "Path": "/opt/entrypoint.sh",
            "Args": [],
            "State": {
                "Status": "running",
                "Running": true,
                "Paused": false,
                "Restarting": false,
                "OOMKilled": false,
                "Dead": false,
                "Pid": 20511,
                "ExitCode": 0,
                "Error": "",
                "StartedAt": "2017-07-21T12:51:37.667083601Z",
                "FinishedAt": "2017-07-21T12:51:37.13950323Z"
            },
            "Image": "sha256:eb82a0e154c810e42bd8f1506696f88e10719b6c5c87ad0aa21d15096c6b6b74",
            "ResolvConfPath": "/var/lib/docker/containers/8151d099cf06795b1e2ffa0ed8f23b3b769e70e8d448d507c2f8962afa06b20a/resolv.conf",
            "HostnamePath": "/var/lib/docker/containers/8151d099cf06795b1e2ffa0ed8f23b3b769e70e8d448d507c2f8962afa06b20a/hostname",
            "HostsPath": "/var/lib/docker/containers/8151d099cf06795b1e2ffa0ed8f23b3b769e70e8d448d507c2f8962afa06b20a/hosts",
            "LogPath": "/var/lib/docker/containers/8151d099cf06795b1e2ffa0ed8f23b3b769e70e8d448d507c2f8962afa06b20a/8151d099cf06795b1e2ffa0ed8f23b3b769e70e8d448d507c2f8962afa06b20a-json.log",
            "Name": "/pmm-server",
            "RestartCount": 0,
            "Driver": "devicemapper",
            "MountLabel": "",
            "ProcessLabel": "",
            "AppArmorProfile": "",
            "ExecIDs": null,
            "HostConfig": {
                "Binds": null,
                "ContainerIDFile": "",
                "LogConfig": {
                    "Type": "json-file",
                    "Config": {}
                },
                "NetworkMode": "default",
                "PortBindings": {
                    "80/tcp": [
                        {
                            "HostIp": "",
                            "HostPort": "80"
                        }
                    ]
                },
                "RestartPolicy": {
                    "Name": "always",
                    "MaximumRetryCount": 0
                },
                "AutoRemove": false,
                "VolumeDriver": "",
                "VolumesFrom": [
                    "pmm-data"
                ],
                "CapAdd": null,
                "CapDrop": null,
                "Dns": [],
                "DnsOptions": [],
                "DnsSearch": [],
                "ExtraHosts": null,
                "GroupAdd": null,
                "IpcMode": "",
                "Cgroup": "",
                "Links": null,
                "OomScoreAdj": 0,
                "PidMode": "",
                "Privileged": false,
                "PublishAllPorts": false,
                "ReadonlyRootfs": false,
                "SecurityOpt": null,
                "UTSMode": "",
                "UsernsMode": "",
                "ShmSize": 67108864,
                "Runtime": "runc",
                "ConsoleSize": [
                    0,
                    0
                ],
                "Isolation": "",
                "CpuShares": 0,
                "Memory": 0,
                "CgroupParent": "",
                "BlkioWeight": 0,
                "BlkioWeightDevice": null,
                "BlkioDeviceReadBps": null,
                "BlkioDeviceWriteBps": null,
                "BlkioDeviceReadIOps": null,
                "BlkioDeviceWriteIOps": null,
                "CpuPeriod": 0,
                "CpuQuota": 0,
                "CpusetCpus": "",
                "CpusetMems": "",
                "Devices": [],
                "DiskQuota": 0,
                "KernelMemory": 0,
                "MemoryReservation": 0,
                "MemorySwap": 0,
                "MemorySwappiness": -1,
                "OomKillDisable": false,
                "PidsLimit": 0,
                "Ulimits": null,
                "CpuCount": 0,
                "CpuPercent": 0,
                "IOMaximumIOps": 0,
                "IOMaximumBandwidth": 0
            },
            "GraphDriver": {
                "Name": "devicemapper",
                "Data": {
                    "DeviceId": "14",
                    "DeviceName": "docker-202:1-528005-536a58e06e4f7533f14030ad1287b38fb34a0370233cbb92b7904be292dd45f0",
                    "DeviceSize": "10737418240"
                }
            },
            "Mounts": [
                {
                    "Name": "127bf9b967abff17998dc64cf09de329e61e865dbfa7a265ae534a5a4ef74326",
                    "Source": "/var/lib/docker/volumes/127bf9b967abff17998dc64cf09de329e61e865dbfa7a265ae534a5a4ef74326/_data",
                    "Destination": "/var/lib/grafana",
                    "Driver": "local",
                    "Mode": "",
                    "RW": true,
                    "Propagation": ""
                },
                {
                    "Name": "9adde5bd070013481c4f70fa61836245c72d286c6389fd6fbc823c21b9fee662",
                    "Source": "/var/lib/docker/volumes/9adde5bd070013481c4f70fa61836245c72d286c6389fd6fbc823c21b9fee662/_data",
                    "Destination": "/var/lib/mysql",
                    "Driver": "local",
                    "Mode": "",
                    "RW": true,
                    "Propagation": ""
                },
                {
                    "Name": "a860793ea7339c7c592ddc03b8f5be114ea0f367f440722f431f7dccd39fc618",
                    "Source": "/var/lib/docker/volumes/a860793ea7339c7c592ddc03b8f5be114ea0f367f440722f431f7dccd39fc618/_data",
                    "Destination": "/opt/consul-data",
                    "Driver": "local",
                    "Mode": "",
                    "RW": true,
                    "Propagation": ""
                },
                {
                    "Name": "88cb8e99bc39144cb2a511afe466e22380e2c3fb96f54f6917ad01f58384f276",
                    "Source": "/var/lib/docker/volumes/88cb8e99bc39144cb2a511afe466e22380e2c3fb96f54f6917ad01f58384f276/_data",
                    "Destination": "/opt/prometheus/data",
                    "Driver": "local",
                    "Mode": "",
                    "RW": true,
                    "Propagation": ""
                }
            ],
            "Config": {
                "Hostname": "8151d099cf06",
                "Domainname": "",
                "User": "",
                "AttachStdin": false,
                "AttachStdout": false,
                "AttachStderr": false,
                "ExposedPorts": {
                    "443/tcp": {},
                    "80/tcp": {}
                },
                "Tty": false,
                "OpenStdin": false,
                "StdinOnce": false,
                "Env": [
                    "METRICS_MEMORY=4194304",
                    "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
                ],
                "Cmd": [
                    "/opt/entrypoint.sh"
                ],
                "Image": "percona/pmm-server:1.2.0",
                "Volumes": null,
                "WorkingDir": "/opt",
                "Entrypoint": null,
                "OnBuild": null,
                "Labels": {
                    "build-date": "20161214",
                    "license": "GPLv2",
                    "name": "CentOS Base Image",
                    "vendor": "CentOS"
                }
            },
            "NetworkSettings": {
                "Bridge": "",
                "SandboxID": "5918ef1e9225c1286002f625168dacff341909b1931756ab2bedc7919a942402",
                "HairpinMode": false,
                "LinkLocalIPv6Address": "",
                "LinkLocalIPv6PrefixLen": 0,
                "Ports": {
                    "443/tcp": null,
                    "80/tcp": [
                        {
                            "HostIp": "0.0.0.0",
                            "HostPort": "80"
                        }
                    ]
                },
                "SandboxKey": "/var/run/docker/netns/5918ef1e9225",
                "SecondaryIPAddresses": null,
                "SecondaryIPv6Addresses": null,
                "EndpointID": "73b31f9d4d274ac894119231b8f9d6124842efdc642fb2b24e263f33dc682a5f",
                "Gateway": "172.17.0.1",
                "GlobalIPv6Address": "",
                "GlobalIPv6PrefixLen": 0,
                "IPAddress": "172.17.0.2",
                "IPPrefixLen": 16,
                "IPv6Gateway": "",
                "MacAddress": "02:42:ac:11:00:02",
                "Networks": {
                    "bridge": {
                        "IPAMConfig": null,
                        "Links": null,
                        "Aliases": null,
                        "NetworkID": "db1d2b01f9f33f3a873f29a2f49bf0dd2bb5ccdbb6490e6180d087e8d0ba12ab",
                        "EndpointID": "73b31f9d4d274ac894119231b8f9d6124842efdc642fb2b24e263f33dc682a5f",
                        "Gateway": "172.17.0.1",
                        "IPAddress": "172.17.0.2",
                        "IPPrefixLen": 16,
                        "IPv6Gateway": "",
                        "GlobalIPv6Address": "",
                        "GlobalIPv6PrefixLen": 0,
                        "MacAddress": "02:42:ac:11:00:02"
                    }
                }
            }
        }
    ]
    
  • MykolaMykola Percona Percona Staff Role
    If you want you can try to clean up prometheus database.
    after this action, you will lose all data collected in Metrics Monitor (in grafana graphs) :(
    I think it should fix high CPU load by prometheus.
    # stop prometheus
    docker exec -it pmm-server supervisorctl stop prometheus
    # backup prometheus database
    docker cp pmm-server:/opt/prometheus/data /tmp/prometheus-bak
    # remove prometheus database
    docker exec -it pmm-server bash -c 'rm -rf /opt/prometheus/data/*'
    # start prometheus
    docker exec -it pmm-server supervisorctl start prometheus
    
Sign In or Register to comment.

MySQL, InnoDB, MariaDB and MongoDB are trademarks of their respective owners.
Copyright ©2005 - 2020 Percona LLC. All rights reserved.