Hi,
I would like to Migrate all the PMM server docker data to new server without any data loss. We are actually Migrating our servers from Centos to Rocky Linux so, we want to migrate PMM Server(Centos) to Rocky Linux new server.
Hi,
I would like to Migrate all the PMM server docker data to new server without any data loss. We are actually Migrating our servers from Centos to Rocky Linux so, we want to migrate PMM Server(Centos) to Rocky Linux new server.
Well, as fate would have it, we’re actually just about to release our PMM Server on a EL9 based image, Oracle Linux 9 specifically.
In order to guide you on the migration we’d need to know a bit about how you’ve installed PMM. Can you provide the following:
du -sh /srv/
should give enough info for that.pmm-admin config
command you run on the clients can help answer this).There may be more info needed but this helps provide the best outline of the steps you’d need to take. The process is pretty straight forward and I’ve been doing it myself for several months now including live customer migrations.
Here are the details:
du -sh /srv/
should give enough info for that. – 43GB
pmm-admin config
command you run on the clients can help answer this). – IP
Please let me know if you need more details on the same.
Ok, it’s pretty straight forward then. I assume you’re going to want to create a parallel system to the existing one so you can have time to install/configure/tune your host OS, get PMM up and running, migrate your data, then perform the final cutover. You can do it all in place too but you’ll lose metrics.
There’s a EL9 based 2.36.0 docker already available you can pull it from perconalab/pmm-server:2.36.0-el9
this has gone through all the same testing and validation that the EL7 container gets and has passed QA.
The steps are:
docker run
you did on the old system only substitute the el9 image over the default el7 one.That’s basically it! Read the steps over and ask any questions…it will help us solidify the instructions we publish!
@steve.hoffman I appreciate your providing such thorough instructions. I’ll try the aforementioned process and report back to you on the results.
I must be missing a step, or something else is going wrong with this process.
Here are the exact steps I’m following:
First, updated to the latest version, and restarted everything.
Backup original to the NFS share “/backups/latest”
sudo docker cp pmm-data:/srv /backups/latest/
Shut off the old server, power on the new server (Apache already configured)
sudo dnf -y install docker-ce docker-ce-cli containerd.io
sudo systemctl enable --now docker containerd
sudo docker create -v /srv --name pmm-data percona/pmm-server:2 /bin/true
sudo docker run -d -p 8080:80 -p 8443:443 --volumes-from pmm-data --name pmm-server --restart always percona/pmm-server:2
sudo systemctl restart httpd
Checked and make sure the initial install works, and it does. I can login with admin/admin
Restore the original data from above:
sudo docker stop pmm-server
sudo docker cp /backups/latest/srv pmm-data:/
sudo docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown -R root:root /srv && \
sudo docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown -R pmm:pmm /srv/alertmanager && \
sudo docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown -R root:pmm /srv/clickhouse && \
sudo docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown -R grafana:grafana /srv/grafana && \
sudo docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown -R pmm:pmm /srv/logs && \
sudo docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown -R postgres:postgres /srv/postgres && \
sudo docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown -R pmm:pmm /srv/prometheus && \
sudo docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown -R pmm:pmm /srv/victoriametrics && \
sudo docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown -R postgres:postgres /srv/logs/postgresql.log
sudo docker start pmm-server
It finishes with:
Successfully copied 6.53GB to pmm-data:/
chown: cannot access ‘/srv/postgres’: No such file or directory
Try to access the app again, and all I get is Nginx’s " 500 Internal Server Error".
in the one of the more recent versions I believe postgres directory name was changed from /srv/postgres
to /srv/postgres14
and I’m guessing the instructions didn’t follow…
so try running it with
sudo docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown -R postgres:postgres /srv/postgres14 && \
Edit: found this in the docs and submitted PR to update for latest versions of PMM
Yeah, already figured that one out, but thank you.
The real issue was with Nginx. Couldn’t figure that one out, so ended up having to leave that alone, and not restore it. So far, everything appears to be working correctly. Here are the new, exact steps I took for the restore part:
sudo docker cp /backups/latest/srv/alerting pmm-data:/srv/
sudo docker cp /backups/latest/srv/alertmanager pmm-data:/srv/
sudo docker cp /backups/latest/srv/clickhouse pmm-data:/srv/
sudo docker cp /backups/latest/srv/grafana pmm-data:/srv/
sudo docker cp /backups/latest/srv/pmm-distribution pmm-data:/srv/
sudo docker cp /backups/latest/srv/postgres14 pmm-data:/srv/
sudo docker cp /backups/latest/srv/prometheus pmm-data:/srv/
sudo docker cp /backups/latest/srv/victoriametrics pmm-data:/srv/
sudo docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown -R root:root /srv && \
sudo docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown -R pmm:pmm /srv/alertmanager && \
sudo docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown -R root:pmm /srv/clickhouse && \
sudo docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown -R grafana:grafana /srv/grafana && \
sudo docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown -R pmm:pmm /srv/logs && \
sudo docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown -R postgres:postgres /srv/postgres14 && \
sudo docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown -R pmm:pmm /srv/prometheus && \
sudo docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown -R pmm:pmm /srv/victoriametrics && \
sudo docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown -R postgres:postgres /srv/logs/postgresql14.log
It’s been running for about 22hrs now, and so far so good.
@steve.hoffman I am getting the below error after following up on the below blog.
[root@test124 log]# docker cp /db/log/srv pmm-data:/
Successfully copied 492kB to pmm-data:/
Error response from daemon: cannot overwrite non-directory “/srv/clickhouse/data/pmm/metrics” with directory “/”
[root@test124 log]#
Can you please help what I am missing here?
Hello there,
Most probably, you already have this directory, you may want to try to check what you already have inside of this dir, to dir inspect the volume:
docker volume inspect pmm-data
Regards,
Denis Subbota.
Managed Services, Percona.
Unanswered | Unsolved | Solved
MySQL, InnoDB, MariaDB and MongoDB are trademarks of their respective owners.
Copyright © 2006 - 2024 Percona LLC. All rights reserved.