I’ve been trying to update our PMM docker container to the newest version. We are currently running PMM 2.9.1. I confirmed that the volume I cannot use the built-in PMM upgrade functionality because our environment is not allowed to use the internet.
My problem is that we can’t seem to migrate/copy the services configuration or exporters. The data from Prometheus is available but the copy of the container does not contain any instance information or any of our Grafana configuration (user accounts, dashboards, etc). I confirmed that the /srv volume can be moved to the newly updated container and the prometheus data can be browsed. PMM isn’t pulling any new information.
I’ve tried copying the entire contents of one container to the other with docker cp, but that didn’t seem to help.
Does anyone have any advice on how to migrate my exporter service configuration? Do I need to rebuild from scratch if I need to update my version?
So to perform the update are you trying to create a container of the newest version alongside the 2.9.1 version and then copy the contents between the two?
Typically we do the following to update containers:
take a backup of the pmm-data container (includes backup of pmm-server by renaming) so you can revert to the old version if you desire
run the same docker run command you originally used only reference the new container image vs the old
so if you previously ran sudo docker run --detach --restart always --publish 80:80 --publish 443:443 --volumes-from pmm-data --name pmm-server percona/pmm-server:2.9.1
you would now run sudo docker run --detach --restart always --publish 80:80 --publish 443:443 --volumes-from pmm-data --name pmm-server percona/pmm-server:2.15.1 (assuming that 2.15.1 is the image you downloaded to your container host.
I’m a little confused what exporter service config you’re referring to since custom queries would be defined on the client side or custom rules would go into the /srv/prometheus and be preserved assuming you created a pmm-data container
So to perform the update are you trying to create a container of the newest version alongside the 2.9.1 version and then copy the contents between the two?
Yes this is my goal.
When I mention the exporter configuration, I’m referring to the inventory of nodes being monitored. When I check the new container, the data that Prometheus has scraped is accessible. However the inventory is empty and the new container is not scraping new data. My thinking is that the inventory configuration is found in /srv but perhaps that’s not true?
I will review your suggestions and get back to you.
I found that I didn’t configure the pmm-data volume correctly. I followed the procedure here to resolve it:
I was able to get the data imported into the updated container (whilst retaining the old container). The monitoring inventory was also retained, I can see all of the nodes I’ve configured in the inventory list.
I have encountered another problem though, which is different than my original issue. Grafana doesn’t seem like it can load a list of nodes, they are listed as ‘N/A’.
I’ll need to dig into this and I’ll create a new topic for the new issue if I can’t resolve it.
Carlos is one smart dude (author of that blog post).
I save myself a ton of headache in troubleshooting the N/A’s by first going to PMM Settings → PMM Inventory and clicking the Services/Agents/Nodes tabs to see if there are any issues…I also use the https:///victoriametrics/targets page to see any glaring issues…sometimes the errors are cryptic at first but tend to make sense if you think about them. If neither of those show any issues start a new topic with whatever info/screenshots you can and we’ll get you squared away!