Hi all,
Per page, Upgrade
I upgrade pmm-server from 2.19 to 2.23.
In summary section, it says upgrade includes four steps: stop container; backup container and data; pull the latest image; and run.
However, in the detail section, after " Run it", it shows “Performa a restore”. In which case, I have to restore after upgrade? I know a little about docker, but I see the mertrics looks okay after upgrade to 2.23. I’m not sure which are missing. Could you please advise?
===============>UPD: The previous metrics were lost … It looks like my upgrade had a problem. So, how to recover metrics after docker image is upgrade?
And one more question, Step 3 “Run it”. Does the operation missing a “-p 80:80”? I tried this and couldn’t access PMM website. Then, I removed the docker container and rerun with “-p 80:80”, it’s okay.
BTW: I created PMM dockers by the following commands:
docker create -v /srv --name pmm-data percona/pmm-server:2.19.0 /bin/true
docker run -d -p 80:80 -p 443:443 --volumes-from pmm-data --name pmm-server --restart always percona/pmm-server:2.19.0
[root@]# docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
percona/pmm-server 2 b031ff61b727 2 weeks ago 1.91GB
percona/pmm-server 2.23.0 b031ff61b727 2 weeks ago 1.91GB
percona/pmm-server 2.19.0 82d29be43d66 4 months ago 1.88GB
Thanks,
Dillon
Hi Dillon,
Could you provide an output of the next command?
docker ps -a
Hi adivinho,
Thank you for advise again.
It looks ok.
I tried to reproduce the issue but 2.19.0 data is shown correct on new pmm-server 2.23.0
e.g.
Could you check VM logs?
docker exec -it pmm-server grep "opened storage" /srv/logs/victoriametrics.log
I’ll post the log message later.
But, I don’t believe this is a bug for upgrade. I’m not sure whether this cause the problem:
In document, it create pmm-data by
docker create -v /srv --name pmm-data percona/pmm-server:2 /bin/true
However, I use this:
docker create -v /srv --name pmm-data percona/pmm-server:2.19.0 /bin/true
Will this “tag” cause the problem?
Thanks,
Dillon
Tag 2 is equal to the latest version. So it was 2.19.0 at the time of creating the container.
It looks like logs have been already rotated.
Please try to check rotated logs.
e.g.
docker exec -it pmm-server grep "opened storage" /srv/logs/victoriametrics.log.1
docker exec -it pmm-server grep "opened storage" /srv/logs/victoriametrics.log.2
docker exec -it pmm-server grep "opened storage" /srv/logs/victoriametrics.log.3
Yes. It have been rotated.
Please see screenshot.
So VM pulled all old metrics after the migration. It had more than 5 millions of rows.
But after pmm-server restart all such data were cleaned up.
Please take a look at my installation screenshot. VM data are constantly increasing.
Maybe you could find out the reason of such behaviour in VM logs.
docker exec -it pmm-server less /srv/logs/victoriametrics.log.3
Unfortunately, victoriametrics.log.3 was rotated and overwrote.
I tried to reproduct this problem but failed:
- stop pmm-server docker container for 2.23.
- remove pmm-server docker by “docker container rm container-id”
- run a new docker by “docker run --detach --restart always -p 80:80 -p 443:443 --volumes-from pmm-data --name pmm-server percona/pmm-server:2.23.0”
- the data still shows in the new PMM webUI.
May this was a mistake when I upgrade 2.19 to 2.23. I’ll check again and try to upgrade 2.23 against prod.
Thanks for advise.
Dillon
Hi adivinho,
Thank for advise above. I tried to upgrade on prod environment. The data is okay.
