Migrating PMM2 data from OVA to Docker.

Percona Monitoring and Management (PMM) PMM 2.x
1

Hello!

I’ve been running PMM2 for a while in a VM provisioned using the PMM2 OVA. I’m looking into moving to Docker now and I’m wondering how best to move the collected data from the existing VM to PMM2 Docker.

Any pointers would be appreciated.

PS: Fairly new to Docker. :slight_smile:

2

Hi Reio!

We don’t have any steps validated for these kinds of migrations, but we can look into it. What PMM version in particular are you using in the OVA setup? Also, do you have any custom configurations in place, or are you using stock PMM? For instance, did you change retention periods, or scrape intervals? Or have you added any custom dashboards or graphs?

Best,

Agustín.

3

Hello!

The OVA VM is currently at 2.10.1.

I haven’t really customised PMM - I’m using it mostly to monitor a few MySQL servers and keep an eye on replication.

I suspect it’s mostly down to getting Prometheus data into the Docker container?

Thanks!

4

Hmmm…this should actually be quite doable since the OVA and Docker container are setup the same inside.

Really the task is to

  1. backup the “data that matters” on PMM2 OVA

  2. move the data to the new location

  3. restore it to the docker version

So that said what you need/care about is in /srv inside your OVA vm so I supposed it’d be:

  1. Get a console on VM and shutdown the services to stop the data from changing
supervisorctl clickhouse grafana nginx prometheus alertmanager pmm-managed
2) tar/gzip your /srv directory
tar -cvf - /srv | gzip >file.tar.gz

  1. scp/sneakernet that file to the new host

  2. Extract the data to the docker pmm-data volume (here’s a good link for that part, see: Restore Into a New pmm-data Mounted in /srv)

  3. connect that data volume to your docker pmm-server as part of the normal “docker run” command.

If your IP/hostname of the pmm-server didn’t change, that’s it, the QAN data should just start arriving at your new pmm-server and your pmm-server will have the internal DB for where to scrape the other exporters. If it did change you’ll need to reregister all the pmm-clients to the new hostname/IP (or just update the /usr/local/percona/pmm2/config/pmm-agent.yaml file with the new hostname and restart pmm-agent)

I would give that a try as a test before doing it for real though!

5

I’ve actually tried the very same. I’ve gotten as far as to log in with credentials from the old VM, but some pages have been giving errors etc. and I haven’t seen any data collected by the old VM. Will keep trying. :slight_smile:

6

If you post more details about errors we can see what we can do to help! I’ve definitely done the backup and restore outlined in the blog post I linked to (created a test instance without the pmm-data container…that turned into more than just test) and so I know they work from that point.

7

It turns out the culprit was using podman instead of docker to do all the work! I’ve now used pure docker and so far so good. I managed to log in with the old credentials and I can see the data.

It seems podman is still some way from being compatible with docker. I think podman cp was the part that worked differently.

Thanks!

8

I see the OVA was based on CentOS 7. Has anyone tried running PMM in a CentOS 7 Linux Container?

9

Migration is now successful, however I notice pmm-managed burning the CPU constantly.

%Cpu0 : 49.5 us, 8.7 sy, 0.0 ni, 38.5 id, 0.3 wa, 1.3 hi, 1.7 si, 0.0 st
%Cpu1 : 44.2 us, 7.1 sy, 0.0 ni, 45.6 id, 0.7 wa, 1.4 hi, 1.0 si, 0.0 st
  PID USER   PR NI  VIRT  RES  SHR S %CPU %MEM   TIME+ COMMAND                                                             
  1877 root   20  0 733888 42892 15984 S 104.0  2.1  2:13.57 pmm-managed
10

Thanks for the heads up. We found it too! We just released pmm 2.11.1 specifically to address this issue (I had to strap my laptop to my desk to keep it from hovering away because my fans were working overtime all weekend!). I verified this version resolves that.

You can use the in place upgrade in the UI or replace the pmm-server container with a 2.11.1 container. Always make sure you follow the backup steps for your pmm-data volume though!

Thanks is for using the app! Please keep the feedback coming!

11

Thanks! I upgraded yesterday and CPU usage is fine now. :slight_smile: However there’s another quirk - probably from migrating the /srv from OVA.

The update to 2.11.1 completed but got stuck in an endless loop until I restarted the container.

TASK [Check volume size] *******************************************************
fatal: [localhost]: FAILED! => {"changed": false, "msg": "Path /var/lib/cloud/scripts/per-boot/resize-xfs does not exist !", "rc": 257}


PLAY RECAP *********************************************************************
localhost                  : ok=12   changed=3    unreachable=0    failed=1    skipped=1    rescued=0    ignored=0


time="2020-10-19T21:49:10Z" level=fatal msg="RunPlaybook failed: exit status 2"

I’m guessing there’s an update related file I should delete in /srv.