Hi,
I would like to Migrate all the PMM server docker data to new server without any data loss. We are actually Migrating our servers from Centos to Rocky Linux so, we want to migrate PMM Server(Centos) to Rocky Linux new server.
Well, as fate would have it, we’re actually just about to release our PMM Server on a EL9 based image, Oracle Linux 9 specifically.
In order to guide you on the migration we’d need to know a bit about how you’ve installed PMM. Can you provide the following:
- How did you install PMM? (Docker, AMI, OVF)
- How is /srv connected to your PMM? (if docker are you using volume, bind mount, or storing in native container)
- How big is your /srv directory?
du -sh /srv/should give enough info for that. - How are your clients registered to the PMM server? By hostname or IP address (the
pmm-admin configcommand you run on the clients can help answer this). - What is your expectation of downtime or recover point objective for the cutover?
- What version of Server and Client(s) are you running?
There may be more info needed but this helps provide the best outline of the steps you’d need to take. The process is pretty straight forward and I’ve been doing it myself for several months now including live customer migrations.
Here are the details:
- How did you install PMM? (Docker, AMI, OVF) – Docker
- How is /srv connected to your PMM? (if docker are you using volume, bind mount, or storing in native container) – bind mount (Seperate Directory Mounted Instead of Default Directory)
- How big is your /srv directory?
du -sh /srv/should give enough info for that. – 43GB - How are your clients registered to the PMM server? By hostname or IP address (the
pmm-admin configcommand you run on the clients can help answer this). – IP - What is your expectation of downtime or recover point objective for the cutover? – No ETA
- What version of Server and Client(s) are you running? – PMM 2.36 Version
Please let me know if you need more details on the same.
Ok, it’s pretty straight forward then. I assume you’re going to want to create a parallel system to the existing one so you can have time to install/configure/tune your host OS, get PMM up and running, migrate your data, then perform the final cutover. You can do it all in place too but you’ll lose metrics.
There’s a EL9 based 2.36.0 docker already available you can pull it from perconalab/pmm-server:2.36.0-el9 this has gone through all the same testing and validation that the EL7 container gets and has passed QA.
The steps are:
- prep your new host (install/configure to your personal or corporate standards)
- get your new PMM server up and running (you’ll use the same
docker runyou did on the old system only substitute the el9 image over the default el7 one. - migrate the data. Couple ways to do this.
- stop both pmm-servers and scp the data from the old host directory to the new one, get your folder permissions correct (step 7 shows the ownerships you’ll need) and start the new pmm-server up.
- I wrote a utility to be run inside the pmm-server docker that allows you to take a hot-backup, copy that artifact and run the same utility to restore…it handles all the internals so may be easier than a bunch of manual steps.
- you need to re-ip your new host since your clients are all bound by IP but for anyone using DNS registrations you can update DNS and metrics will start flowing once the DNS TTL expires.
That’s basically it! Read the steps over and ask any questions…it will help us solidify the instructions we publish!
@steve.hoffman I appreciate your providing such thorough instructions. I’ll try the aforementioned process and report back to you on the results.
I must be missing a step, or something else is going wrong with this process.
Here are the exact steps I’m following:
First, updated to the latest version, and restarted everything.
Backup original to the NFS share “/backups/latest”
sudo docker cp pmm-data:/srv /backups/latest/
Shut off the old server, power on the new server (Apache already configured)
sudo dnf -y install docker-ce docker-ce-cli containerd.io
sudo systemctl enable --now docker containerd
sudo docker create -v /srv --name pmm-data percona/pmm-server:2 /bin/true
sudo docker run -d -p 8080:80 -p 8443:443 --volumes-from pmm-data --name pmm-server --restart always percona/pmm-server:2
sudo systemctl restart httpd
Checked and make sure the initial install works, and it does. I can login with admin/admin
Restore the original data from above:
sudo docker stop pmm-server
sudo docker cp /backups/latest/srv pmm-data:/
sudo docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown -R root:root /srv && \
sudo docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown -R pmm:pmm /srv/alertmanager && \
sudo docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown -R root:pmm /srv/clickhouse && \
sudo docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown -R grafana:grafana /srv/grafana && \
sudo docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown -R pmm:pmm /srv/logs && \
sudo docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown -R postgres:postgres /srv/postgres && \
sudo docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown -R pmm:pmm /srv/prometheus && \
sudo docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown -R pmm:pmm /srv/victoriametrics && \
sudo docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown -R postgres:postgres /srv/logs/postgresql.log
sudo docker start pmm-server
It finishes with:
Successfully copied 6.53GB to pmm-data:/
chown: cannot access ‘/srv/postgres’: No such file or directory
Try to access the app again, and all I get is Nginx’s " 500 Internal Server Error".
in the one of the more recent versions I believe postgres directory name was changed from /srv/postgres to /srv/postgres14 and I’m guessing the instructions didn’t follow…
so try running it with
sudo docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown -R postgres:postgres /srv/postgres14 && \
Edit: found this in the docs and submitted PR to update for latest versions of PMM
Yeah, already figured that one out, but thank you.
The real issue was with Nginx. Couldn’t figure that one out, so ended up having to leave that alone, and not restore it. So far, everything appears to be working correctly. Here are the new, exact steps I took for the restore part:
sudo docker cp /backups/latest/srv/alerting pmm-data:/srv/
sudo docker cp /backups/latest/srv/alertmanager pmm-data:/srv/
sudo docker cp /backups/latest/srv/clickhouse pmm-data:/srv/
sudo docker cp /backups/latest/srv/grafana pmm-data:/srv/
sudo docker cp /backups/latest/srv/pmm-distribution pmm-data:/srv/
sudo docker cp /backups/latest/srv/postgres14 pmm-data:/srv/
sudo docker cp /backups/latest/srv/prometheus pmm-data:/srv/
sudo docker cp /backups/latest/srv/victoriametrics pmm-data:/srv/
sudo docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown -R root:root /srv && \
sudo docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown -R pmm:pmm /srv/alertmanager && \
sudo docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown -R root:pmm /srv/clickhouse && \
sudo docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown -R grafana:grafana /srv/grafana && \
sudo docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown -R pmm:pmm /srv/logs && \
sudo docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown -R postgres:postgres /srv/postgres14 && \
sudo docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown -R pmm:pmm /srv/prometheus && \
sudo docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown -R pmm:pmm /srv/victoriametrics && \
sudo docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown -R postgres:postgres /srv/logs/postgresql14.log
It’s been running for about 22hrs now, and so far so good.
@steve.hoffman I am getting the below error after following up on the below blog.
[root@test124 log]# docker cp /db/log/srv pmm-data:/
Successfully copied 492kB to pmm-data:/
Error response from daemon: cannot overwrite non-directory “/srv/clickhouse/data/pmm/metrics” with directory “/”
[root@test124 log]#
Can you please help what I am missing here?