Could not start PMM server after update version from 1.5.3 to 1.8

Percona Monitoring and Management (PMM) PMM 1.x
1

Dear Supporters,

Dear hrvojem,
So great and thank you for information about new latest version of PMM ! I’ve just update new latest version of PMM server. But unfortunately, I could not restart my PMM server. I’ve tried re-install completely docker and PMM also. But i got same issue like this:

[root@test ~]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
681746cc6f9f percona/pmm-server:latest “/opt/entrypoint.sh” 36 seconds ago Restarting (1) 7 seconds ago 0.0.0.0:80->80/tcp, 443/tcp pmm-server

Could you please help me troubleshooting this issue ?

Thank you for your contribution ! Hope you response to me as soon as possible.

2

Hi khuent!

Please could you provide some more information:

  • docker logs pmm-server
  • docker inspect pmm-server --format ‘{{ .Mounts }}’
  • details about the process that you went through to upgrade/install the container

UPDATE: I have tried 2 approaches for upgrade and on both occasions still have a running instance afetrwards

  • using the upgrade functionality in PMM
  • creating a new container for 1.8.0 and using the volumes created by 1.5.3

Presuming that the setup matches the online documentation ([url]Percona Monitoring and Management) then you could try something like the following approach to get inside the container:

$ docker run -it --entrypoint /bin/bash -d -p 80:80 --volumes-from pmm-data --name pmm-server --restart always percona/pmm-server:1.8.0
$ docker exec -it pmm-server /bin/bash

[root@33ac27da72f0 opt]# ( /opt/entrypoint.sh >debug.log 2>&1 ) &
[root@33ac27da72f0 opt]# tail -f debug.log

[root@33ac27da72f0 opt]# supervisorctl status
consul RUNNING pid 254, uptime 0:04:30
createdb EXITED Mar 02 12:55 PM
createdb2 EXITED Mar 02 12:55 PM
createdb3 EXITED Mar 02 12:55 PM
cron RUNNING pid 257, uptime 0:04:30
dashboard-upgrade EXITED Mar 02 12:55 PM
grafana RUNNING pid 255, uptime 0:04:30
mysql RUNNING pid 253, uptime 0:04:30
nginx RUNNING pid 256, uptime 0:04:30
node_exporter RUNNING pid 277, uptime 0:04:30
orchestrator STOPPED Not started
pmm-manage RUNNING pid 279, uptime 0:04:30
pmm-managed RUNNING pid 359, uptime 0:04:29
prometheus RUNNING pid 264, uptime 0:04:30
qan-api RUNNING pid 259, uptime 0:04:30

Thanks

Ceri

3

Thank you for your support. I’ve tried to upgrade to 1.7 already. It’s running perfectly. I will plan upgrade from 1.7 to 1.8. And i will give you information as you instruct to me. Before I keep running version 1.7, I’ve tried upgrade from 1.7 to 1.8 but it failed same as I posted in this post. My upgrading progress following below:
1/ Stop docker old container (pmm-server)
2/ Rename current container (pmm-server)
3/ Pull new image
4/ Start docker with --restart option

I have question: Could I keep running old version pmm-client with new version pmm-server or we need upgrade both same version? Could I need to stop all pmm-client before upgrade pmm-server?

Thank you so much !

KhueNT

4

Dear Ceri,

I’ve just upgraded my PMM-server up to 1.8.1 from 1.7 but I’ve got the same issue.
Firstly, I did follow as the link you give me, the issue appeared. Then I execute the command you instruct me:
docker run -it --entrypoint /bin/bash -d -p 80:80 --volumes-from pmm-data --name pmm-server --restart always percona/pmm-server:1.8.0 It’s running like this :

[root@bitest ~]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ebc48b121cc6 percona/pmm-server:latest “/bin/bash” 3 seconds ago Up 2 seconds 0.0.0.0:80->80/tcp, 443/tcp pmm-server

But I could not connect and access from browser.

There are some logs that you need:

[root@bitest ~]# docker logs pmm-server
[root@bitest ~]# docker inspect pmm-server --format ‘{{ .Mounts }}’
Error: No such image or container: --format
Error: No such image or container: {{ .Mounts }}
[
{
“Id”: “ebc48b121cc6e3c64c392bae2ecb1e1a0f60136213707adbace59722619a0d40”,
“Created”: “2018-03-08T04:15:04.290230544Z”,
“Path”: “/bin/bash”,
“Args”: ,
“State”: {
“Running”: true,
“Paused”: false,
“Restarting”: false,
“OOMKilled”: false,
“Dead”: false,
“Pid”: 1342,
“ExitCode”: 0,
“Error”: “”,
“StartedAt”: “2018-03-08T04:15:05.447259347Z”,
“FinishedAt”: “0001-01-01T00:00:00Z”
},
“Image”: “de562dadc7213847c60cf0cee2336f8520760bce4303046413c8c044fd7b0369”,
“NetworkSettings”: {
“Bridge”: “”,
“EndpointID”: “972fd4b017031eeca7b9ab6f6176b79c5ab1977be9dbc7f3dd4764491021466b”,
“Gateway”: “172.x.x.x”,
“GlobalIPv6Address”: “”,
“GlobalIPv6PrefixLen”: 0,
“HairpinMode”: false,
“IPAddress”: “172.x.x.x”,
“IPPrefixLen”: 16,
“IPv6Gateway”: “”,
“LinkLocalIPv6Address”: “”,
“LinkLocalIPv6PrefixLen”: 0,
“MacAddress”: “02:42:ac:11:00:06”,
“NetworkID”: “c2467847c4bcbb5144594edaf1991b3ebc7fb25e5961ba544efad5e435ebf9da”,
“PortMapping”: null,
“Ports”: {
“443/tcp”: null,
“80/tcp”: [
{
“HostIp”: “0.0.0.0”,
“HostPort”: “80”
}
]
},
“SandboxKey”: “/var/run/docker/netns/ebc48b121cc6”,
“SecondaryIPAddresses”: null,
“SecondaryIPv6Addresses”: null
},
“ResolvConfPath”: “/var/lib/docker/containers/ebc48b121cc6e3c64c392bae2ecb1e1a0f60136213707adbace59722619a0d40/resolv.conf”,
“HostnamePath”: “/var/lib/docker/containers/ebc48b121cc6e3c64c392bae2ecb1e1a0f60136213707adbace59722619a0d40/hostname”,
“HostsPath”: “/var/lib/docker/containers/ebc48b121cc6e3c64c392bae2ecb1e1a0f60136213707adbace59722619a0d40/hosts”,
“LogPath”: “/var/lib/docker/containers/ebc48b121cc6e3c64c392bae2ecb1e1a0f60136213707adbace59722619a0d40/ebc48b121cc6e3c64c392bae2ecb1e1a0f60136213707adbace59722619a0d40-json.log”,
“Name”: “/pmm-server”,
“RestartCount”: 0,
“Driver”: “devicemapper”,
“ExecDriver”: “native-0.2”,
“MountLabel”: “”,
“ProcessLabel”: “”,
“Volumes”: {
“/opt/consul-data”: “/var/lib/docker/volumes/9844b01b76d72194fa6c7fa5290f6ae54af6cf1ad054916142f3f619450bdc56/_data”,
“/opt/grafana/data”: “/var/lib/docker/volumes/44530acc1449ba156e4d0cc42123fe32f5568e583e2c126422b2ecad95cb450b/_data”,
“/opt/prometheus/data”: “/var/lib/docker/volumes/8368026b25d99ee6309f79b070489a138cd053a4c0e1efc86f4a9b8a08f471ff/_data”,
“/var/lib/mysql”: “/var/lib/docker/volumes/100bde7e9fe233c121c57a0e96c5110c9de37c25cfbe41520fe80337b0efdfc9/_data”
},
“VolumesRW”: {
“/opt/consul-data”: true,
“/opt/grafana/data”: true,
“/opt/prometheus/data”: true,
“/var/lib/mysql”: true
},
“AppArmorProfile”: “”,
“ExecIDs”: null,
“HostConfig”: {
“Binds”: null,
“ContainerIDFile”: “”,
“LxcConf”: ,
“Memory”: 0,
“MemorySwap”: 0,
“CpuShares”: 0,
“CpuPeriod”: 0,
“CpusetCpus”: “”,
“CpusetMems”: “”,
“CpuQuota”: 0,
“BlkioWeight”: 0,
“OomKillDisable”: false,
“Privileged”: false,
“PortBindings”: {
“80/tcp”: [
{
“HostIp”: “”,
“HostPort”: “80”
}
]
},
“Links”: null,
“PublishAllPorts”: false,
“Dns”: null,
“DnsSearch”: null,
“ExtraHosts”: null,
“VolumesFrom”: [
“pmm-data”
],
“Devices”: ,
“NetworkMode”: “bridge”,
“IpcMode”: “”,
“PidMode”: “”,
“UTSMode”: “”,
“CapAdd”: null,
“CapDrop”: null,
“RestartPolicy”: {
“Name”: “always”,
“MaximumRetryCount”: 0
},
“SecurityOpt”: null,
“ReadonlyRootfs”: false,
“Ulimits”: null,
“LogConfig”: {
“Type”: “json-file”,
“Config”: {}
},
“CgroupParent”: “”
},
“Config”: {
“Hostname”: “ebc48b121cc6”,
“Domainname”: “”,
“User”: “”,
“AttachStdin”: false,
“AttachStdout”: false,
“AttachStderr”: false,
“PortSpecs”: null,
“ExposedPorts”: {
“443/tcp”: {},
“80/tcp”: {}
},
“Tty”: true,
“OpenStdin”: true,
“StdinOnce”: false,
“Env”: [
“PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin”
],
“Cmd”: null,
“Image”: “percona/pmm-server:latest”,
“Volumes”: null,
“VolumeDriver”: “”,
“WorkingDir”: “/opt”,
“Entrypoint”: [
“/bin/bash”
],
“NetworkDisabled”: false,
“MacAddress”: “”,
“OnBuild”: null,
“Labels”: {
“build-date”: “20180302”,
“license”: “GPLv2”,
“name”: “CentOS Base Image”,
“vendor”: “CentOS”
}
}
}
]
[root@bitest ~]# docker exec -it pmm-server /bin/bash
[root@ebc48b121cc6 opt]# ( /opt/entrypoint.sh >debug.log 2>&1 ) &
[1] 26
[root@ebc48b121cc6 opt]# tail -f debug.log
cat: /sys/fs/cgroup/memory/memory.limit_in_bytes: No such file or directory

Could you help me out ? and Would you please answer my previous questions ?

Thank you for your support !

5

Hi khuent

Thanks for the output - I notice a few things here, one of which tells you why it is failing.
[LIST]
[*]which version of docker are you using? the command that I provided should not fail

⇒ docker inspect pmm-server --format '{{ .Mounts }}'
	[{volume pmm-prometheus-153 /var/lib/docker/volumes/pmm-prometheus-153/_data /opt/prometheus/data local z true } {volume pmm-grafana-153 /var/lib/docker/volumes/pmm-grafana-153/_data /var/lib/grafana local z true } {volume pmm-mysql-153 /var/lib/docker/volumes/pmm-mysql-153/_data /var/lib/mysql local z true } {volume pmm-consul-153 /var/lib/docker/volumes/pmm-consul-153/_data /opt/consul-data local z true }]

As noted in the documenation, you should have at least 1.12.6. You can see the version with

docker version

[*]the issue looks likely to be related to the docker version, because the PMM entrypoint will exit on any error and you have:

[B]cat: /sys/fs/cgroup/memory/memory.limit_in_bytes: No such file or directory[/B]

[/LIST] Regarding your question about the clients, you should match their version to the one for the running PMM server - when upgrading, I would suggest upgrading the clients as soon as the container has been upgraded.

In addition to the version of Docker, please could you confirm the OS/distribution that you are running Docker on.

Kind regards,

Ceri

6

Hi,

same problem here. I’m running the latest docker-ce edition (17.12.1-ce) on Debian Jessie.

7

Thanks magenbrot for the extra information. We will look into this further for you