Install PMM2 2.30 docker server with error

Percona Monitoring and Management (PMM) PMM 2.x
1

Hi, all

Good morning. I install pmm2 2.30 on a new server with error shows Grafana failed to load application files. Please see screenshot.
image

I check docker image status. It’s healthy.
image

And then I tried to install 2.19, and then upgrade to 2.23 and 2.30. Both 2.19 and 2.23 work. But, 2.30 is failed. The following is my steps:

docker create -v /srv --name pmm-data percona/pmm-server:2.19.0 /bin/true
docker run  -d -p 80:80 -p 443:443 --volumes-from pmm-data --name pmm-server --restart always percona/pmm-server:2.19.0
docker stop pmm-server
docker rename pmm-server pmm-server-219
docker run -d -p 80:80 -p 443:443 --volumes-from pmm-data --name pmm-server --restart always percona/pmm-server:2.23.0
docker stop pmm-server
docker rename pmm-server pmm-server-223
docker run -d -p 443:443 --volumes-from pmm-data --name pmm-server --restart always percona/pmm-server:2.30.0

Do I miss something or something is wrong?

Thanks,
Dillon

1 Like
2

Did you follow these steps?

1 Like
3

Your steps look good for performing upgrades :+1:

Looks like grafana isn’t starting correctly, or nginx isn’t doing proper redirection. I’d like to ask you to investigate via the log files. You can either export them and attach to this issue and we’ll review it, or you can take a look yourself. Here are example steps:

docker exec -it pmm-server bash
cd /srv/logs

You will want to examine the logs nginx and grafana for any interesting events.

1 Like
4

Thank you for reply.

At the beginning, I want to create a new PMM monitor server on a new PC server. But, I can’t access the Web site of 2.30. So, I import 2.19 and 2.23 to check if the docker service is okay. Since I upgraded 2.19 to 2.23 last year. I installed same docker rpm packages and typed the same commands to start PMM service. Soooo, I guess the steps are okay. I post steps in attachment. Could you please verify?

Thanks,
Dillon

1 Like
5

Hi Michael,

Thank you for reply. I take a look the log files. I saw some error in ngixn.log. But it’s generated by pmm-agent. I’m not sure if this is a problem. Could you please help me check the attachments?

Thanks,
Dillon
nginx.log (13.8 KB)
grafana.log (6.8 KB)

1 Like
6

Hi @Dillonlu thanks for sharing your logs. I took a look and agree with you, there doesn’t seem to be any issue clearly apparent from these sources.

I’d suggest two things in order to recover your PMM Server:

  1. Have you tried restarting the container? Sometimes this enables grafana and nginx to come up cleanly
  2. Can you try reverting to the last known working version? in your case it seems that it would be 2.23. You can do this by essentially following the same upgrade steps, however when doing the docker run you’ll reference the container pmm-server-223 instead

Let us know if either of these options gets you back to a running state. If only #2 works then perhaps we can try an upgrade from 2.23 to say 2.29 and verify that version works, and then jump one more time to 2.30 while :crossed_fingers:

1 Like
7

Hi Michael,

I tried all PMM2 version. 2.26 is fine; and 2.27 doesn’t work. Please grafana.log. 226 means PMM2 v2.26.
I compared both grafana.log and nginx.log. It looks like the grafana has some problems. Please check attachments.
nginx227.log (16.2 KB)
grafana226.log (8.9 KB)
grafana227.log (13.9 KB)
nginx226.log (329.3 KB)

At the beginning, there are “Starting Grafana” in grafana226.log. But I don’t see this in granfana227.log. So, I guess something wrong with grafana.

An interesting thing: we installed a VM server of centos 7.9. It ran the 2.30 without error. Perviously, we use RHEL7.9. We don’t know what’s the difference. :rofl: Docker was upgrade to 20.10.18

Thanks,
Dillon

1 Like