Not the answer you need?
Register and ask your own question!

Could not start PMM server after update version from 1.5.3 to 1.8

khuentkhuent EntrantCurrent User Role Supporter
Dear Supporters,

Dear hrvojem,
So great and thank you for information about new latest version of PMM ! I've just update new latest version of PMM server. But unfortunately, I could not restart my PMM server. I've tried re-install completely docker and PMM also. But i got same issue like this:

[[email protected] ~]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
681746cc6f9f percona/pmm-server:latest "/opt/entrypoint.sh" 36 seconds ago Restarting (1) 7 seconds ago 0.0.0.0:80->80/tcp, 443/tcp pmm-server

Could you please help me troubleshooting this issue ?

Thank you for your contribution ! Hope you response to me as soon as possible.

Comments

  • cericeri Percona Percona Staff Role
    Hi khuent!

    Please could you provide some more information:

    - docker logs pmm-server
    - docker inspect pmm-server --format '{{ .Mounts }}'
    - details about the process that you went through to upgrade/install the container

    UPDATE: I have tried 2 approaches for upgrade and on both occasions still have a running instance afetrwards

    - using the upgrade functionality in PMM
    - creating a new container for 1.8.0 and using the volumes created by 1.5.3

    Presuming that the setup matches the online documentation (https://www.percona.com/doc/percona-monitoring-and-management/deploy/server/docker.setting-up.html) then you could try something like the following approach to get inside the container:
    $ docker run -it --entrypoint /bin/bash -d -p 80:80 --volumes-from pmm-data --name pmm-server --restart always percona/pmm-server:1.8.0
    $ docker exec -it pmm-server /bin/bash
    
    [root@33ac27da72f0 opt]# ( /opt/entrypoint.sh >debug.log 2>&1 ) &
    [root@33ac27da72f0 opt]# tail -f debug.log
    
    [root@33ac27da72f0 opt]# supervisorctl status
    consul                           RUNNING   pid 254, uptime 0:04:30
    createdb                         EXITED    Mar 02 12:55 PM
    createdb2                        EXITED    Mar 02 12:55 PM
    createdb3                        EXITED    Mar 02 12:55 PM
    cron                             RUNNING   pid 257, uptime 0:04:30
    dashboard-upgrade                EXITED    Mar 02 12:55 PM
    grafana                          RUNNING   pid 255, uptime 0:04:30
    mysql                            RUNNING   pid 253, uptime 0:04:30
    nginx                            RUNNING   pid 256, uptime 0:04:30
    node_exporter                    RUNNING   pid 277, uptime 0:04:30
    orchestrator                     STOPPED   Not started
    pmm-manage                       RUNNING   pid 279, uptime 0:04:30
    pmm-managed                      RUNNING   pid 359, uptime 0:04:29
    prometheus                       RUNNING   pid 264, uptime 0:04:30
    qan-api                          RUNNING   pid 259, uptime 0:04:30
    

    Thanks

    Ceri
  • khuentkhuent Entrant Current User Role Supporter
    Thank you for your support. I've tried to upgrade to 1.7 already. It's running perfectly. I will plan upgrade from 1.7 to 1.8. And i will give you information as you instruct to me. Before I keep running version 1.7, I've tried upgrade from 1.7 to 1.8 but it failed same as I posted in this post. My upgrading progress following below:
    1/ Stop docker old container (pmm-server)
    2/ Rename current container (pmm-server)
    3/ Pull new image
    4/ Start docker with --restart option

    I have question: Could I keep running old version pmm-client with new version pmm-server or we need upgrade both same version? Could I need to stop all pmm-client before upgrade pmm-server?

    Thank you so much !

    KhueNT
  • khuentkhuent Entrant Current User Role Supporter
    Dear Ceri,

    I've just upgraded my PMM-server up to 1.8.1 from 1.7 but I've got the same issue.
    Firstly, I did follow as the link you give me, the issue appeared. Then I execute the command you instruct me:
    docker run -it --entrypoint /bin/bash -d -p 80:80 --volumes-from pmm-data --name pmm-server --restart always percona/pmm-server:1.8.0 It's running like this :

    [[email protected] ~]# docker ps
    CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
    ebc48b121cc6 percona/pmm-server:latest "/bin/bash" 3 seconds ago Up 2 seconds 0.0.0.0:80->80/tcp, 443/tcp pmm-server

    But I could not connect and access from browser.

    There are some logs that you need:

    [[email protected] ~]# docker logs pmm-server
    [[email protected] ~]# docker inspect pmm-server --format '{{ .Mounts }}'
    Error: No such image or container: --format
    Error: No such image or container: {{ .Mounts }}
    [
    {
    "Id": "ebc48b121cc6e3c64c392bae2ecb1e1a0f60136213707adbace59722619a0d40",
    "Created": "2018-03-08T04:15:04.290230544Z",
    "Path": "/bin/bash",
    "Args": [],
    "State": {
    "Running": true,
    "Paused": false,
    "Restarting": false,
    "OOMKilled": false,
    "Dead": false,
    "Pid": 1342,
    "ExitCode": 0,
    "Error": "",
    "StartedAt": "2018-03-08T04:15:05.447259347Z",
    "FinishedAt": "0001-01-01T00:00:00Z"
    },
    "Image": "de562dadc7213847c60cf0cee2336f8520760bce4303046413c8c044fd7b0369",
    "NetworkSettings": {
    "Bridge": "",
    "EndpointID": "972fd4b017031eeca7b9ab6f6176b79c5ab1977be9dbc7f3dd4764491021466b",
    "Gateway": "172.x.x.x",
    "GlobalIPv6Address": "",
    "GlobalIPv6PrefixLen": 0,
    "HairpinMode": false,
    "IPAddress": "172.x.x.x",
    "IPPrefixLen": 16,
    "IPv6Gateway": "",
    "LinkLocalIPv6Address": "",
    "LinkLocalIPv6PrefixLen": 0,
    "MacAddress": "02:42:ac:11:00:06",
    "NetworkID": "c2467847c4bcbb5144594edaf1991b3ebc7fb25e5961ba544efad5e435ebf9da",
    "PortMapping": null,
    "Ports": {
    "443/tcp": null,
    "80/tcp": [
    {
    "HostIp": "0.0.0.0",
    "HostPort": "80"
    }
    ]
    },
    "SandboxKey": "/var/run/docker/netns/ebc48b121cc6",
    "SecondaryIPAddresses": null,
    "SecondaryIPv6Addresses": null
    },
    "ResolvConfPath": "/var/lib/docker/containers/ebc48b121cc6e3c64c392bae2ecb1e1a0f60136213707adbace59722619a0d40/resolv.conf",
    "HostnamePath": "/var/lib/docker/containers/ebc48b121cc6e3c64c392bae2ecb1e1a0f60136213707adbace59722619a0d40/hostname",
    "HostsPath": "/var/lib/docker/containers/ebc48b121cc6e3c64c392bae2ecb1e1a0f60136213707adbace59722619a0d40/hosts",
    "LogPath": "/var/lib/docker/containers/ebc48b121cc6e3c64c392bae2ecb1e1a0f60136213707adbace59722619a0d40/ebc48b121cc6e3c64c392bae2ecb1e1a0f60136213707adbace59722619a0d40-json.log",
    "Name": "/pmm-server",
    "RestartCount": 0,
    "Driver": "devicemapper",
    "ExecDriver": "native-0.2",
    "MountLabel": "",
    "ProcessLabel": "",
    "Volumes": {
    "/opt/consul-data": "/var/lib/docker/volumes/9844b01b76d72194fa6c7fa5290f6ae54af6cf1ad054916142f3f619450bdc56/_data",
    "/opt/grafana/data": "/var/lib/docker/volumes/44530acc1449ba156e4d0cc42123fe32f5568e583e2c126422b2ecad95cb450b/_data",
    "/opt/prometheus/data": "/var/lib/docker/volumes/8368026b25d99ee6309f79b070489a138cd053a4c0e1efc86f4a9b8a08f471ff/_data",
    "/var/lib/mysql": "/var/lib/docker/volumes/100bde7e9fe233c121c57a0e96c5110c9de37c25cfbe41520fe80337b0efdfc9/_data"
    },
    "VolumesRW": {
    "/opt/consul-data": true,
    "/opt/grafana/data": true,
    "/opt/prometheus/data": true,
    "/var/lib/mysql": true
    },
    "AppArmorProfile": "",
    "ExecIDs": null,
    "HostConfig": {
    "Binds": null,
    "ContainerIDFile": "",
    "LxcConf": [],
    "Memory": 0,
    "MemorySwap": 0,
    "CpuShares": 0,
    "CpuPeriod": 0,
    "CpusetCpus": "",
    "CpusetMems": "",
    "CpuQuota": 0,
    "BlkioWeight": 0,
    "OomKillDisable": false,
    "Privileged": false,
    "PortBindings": {
    "80/tcp": [
    {
    "HostIp": "",
    "HostPort": "80"
    }
    ]
    },
    "Links": null,
    "PublishAllPorts": false,
    "Dns": null,
    "DnsSearch": null,
    "ExtraHosts": null,
    "VolumesFrom": [
    "pmm-data"
    ],
    "Devices": [],
    "NetworkMode": "bridge",
    "IpcMode": "",
    "PidMode": "",
    "UTSMode": "",
    "CapAdd": null,
    "CapDrop": null,
    "RestartPolicy": {
    "Name": "always",
    "MaximumRetryCount": 0
    },
    "SecurityOpt": null,
    "ReadonlyRootfs": false,
    "Ulimits": null,
    "LogConfig": {
    "Type": "json-file",
    "Config": {}
    },
    "CgroupParent": ""
    },
    "Config": {
    "Hostname": "ebc48b121cc6",
    "Domainname": "",
    "User": "",
    "AttachStdin": false,
    "AttachStdout": false,
    "AttachStderr": false,
    "PortSpecs": null,
    "ExposedPorts": {
    "443/tcp": {},
    "80/tcp": {}
    },
    "Tty": true,
    "OpenStdin": true,
    "StdinOnce": false,
    "Env": [
    "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
    ],
    "Cmd": null,
    "Image": "percona/pmm-server:latest",
    "Volumes": null,
    "VolumeDriver": "",
    "WorkingDir": "/opt",
    "Entrypoint": [
    "/bin/bash"
    ],
    "NetworkDisabled": false,
    "MacAddress": "",
    "OnBuild": null,
    "Labels": {
    "build-date": "20180302",
    "license": "GPLv2",
    "name": "CentOS Base Image",
    "vendor": "CentOS"
    }
    }
    }
    ]
    [[email protected] ~]# docker exec -it pmm-server /bin/bash
    [[email protected] opt]# ( /opt/entrypoint.sh >debug.log 2>&1 ) &
    [1] 26
    [[email protected] opt]# tail -f debug.log
    cat: /sys/fs/cgroup/memory/memory.limit_in_bytes: No such file or directory

    Could you help me out ? and Would you please answer my previous questions ?

    Thank you for your support !
  • cericeri Percona Percona Staff Role
    Hi khuent

    Thanks for the output - I notice a few things here, one of which tells you why it is failing.
    • which version of docker are you using? the command that I provided should not fail
      ⇒ docker inspect pmm-server --format '{{ .Mounts }}'
      	[{volume pmm-prometheus-153 /var/lib/docker/volumes/pmm-prometheus-153/_data /opt/prometheus/data local z true } {volume pmm-grafana-153 /var/lib/docker/volumes/pmm-grafana-153/_data /var/lib/grafana local z true } {volume pmm-mysql-153 /var/lib/docker/volumes/pmm-mysql-153/_data /var/lib/mysql local z true } {volume pmm-consul-153 /var/lib/docker/volumes/pmm-consul-153/_data /opt/consul-data local z true }]
      
      As noted in the documenation, you should have at least 1.12.6. You can see the version with
      docker version
      
    • the issue looks likely to be related to the docker version, because the PMM entrypoint will exit on any error and you have:
      [B]cat: /sys/fs/cgroup/memory/memory.limit_in_bytes: No such file or directory[/B]
      
    Regarding your question about the clients, you should match their version to the one for the running PMM server - when upgrading, I would suggest upgrading the clients as soon as the container has been upgraded.

    In addition to the version of Docker, please could you confirm the OS/distribution that you are running Docker on.

    Kind regards,

    Ceri
  • magenbrotmagenbrot Contributor Current User Role Novice
    Hi,
    ceri wrote: »
    the issue looks likely to be related to the docker version, because the PMM entrypoint will exit on any error and you have:
    [B]cat: /sys/fs/cgroup/memory/memory.limit_in_bytes: No such file or directory[/B]
    

    same problem here. I'm running the latest docker-ce edition (17.12.1-ce) on Debian Jessie.
  • cericeri Percona Percona Staff Role
    Thanks magenbrot for the extra information. We will look into this further for you
Sign In or Register to comment.

MySQL, InnoDB, MariaDB and MongoDB are trademarks of their respective owners.
Copyright ©2005 - 2020 Percona LLC. All rights reserved.