I would like to change the path to the bind mounted volume in PMM docker setup. e.g. /mnt/data/srv.
I went through the docker setup and restore procedures. I didn’t quite get the issue with hard requirements on the bind mounted volume name /srv on the persistent container.
PMM Server expects the data volume to be /srv. Using any other value will result in data loss when upgrading.
Can’t I just backup the old one and restore it after upgrading ? or is there anything hidden underneath that I didn’t know of ?
“inside” the running container, PMM expects /srv to contain all the data. You cannot change this. What you can do is bind-mount an external path when creating the primary PMM server container. Example:
docker -v /path/to/somewhere:/srv -p 443:443 percona/pmm-server:2
2 Likes
@matthewb I see. I get it now. Thank you
1 Like
@matthewb Doesn’t that means there will be duplicated data ? i.e, the pmm-data container keeping the data on /var/lib/docker/… and the path where I mount ?
1 Like
No. The core container data will be in /var/lib/docker with all your other containers and volumes but the data in /srv which you bind-mount to /path/where/ever will only be stored there.
I suggest you find some Docker 101 videos and read up on documentation of Docker as a lot of this is basic docker concepts.
1 Like
@matthewb I understand 101 and how the docker works. What I do not understand is the double indirection with the volume mapping. The data in pmm-data which is mapped like so
source: /var/lib/docker/volumes/*/_data
target: /srv
in the pmm-server again I am mapping (like you suggested) to an external volume. Now I have data on the partition where docker is installed, The same data is now being used by the pmm-server. (with --volume-from). I am mapping again to the same volume from the pmm-server to external volume i.e, /path/whatever:/srv The same data is now synced to the external volume on the /path/whatever ? .is my understanding correct here ? would you care to explain please.
1 Like
If you use a bind-mount to the pmm-server container, like I demonstrated above, then pmm-data is irrelevant as you won’t use it at all. Thus, no duplication of data anywhere.
There are 2 exclusive choices:
- use pmm-data container, which writes its data to your docker volumes path
- no pmm-data container and bind-mount path to pmm-server to pick whereever you want the data written
1 Like
@matthewb Exactly that was what I was missing. I had no clue that these two were mutually exclusive.I was thinking that it was done so that to circumvent the user account permission issues in mounted volumes. It was not mentioned in the documentation. Thank you and I really appreciate your time to explain.
1 Like
Just curious here, when I try to bind mount the percona image I found my /srv folder is empty.
If I run:
docker run -v /mnt/myhostmount:/srv -p 443:443 percona/pmm-server:2
I get an empty dir where mnt/myhostmount has nothing and of course since there’s no config, nothing works. I will mention that I have taken the image and further added a few things to it to customize it for my own use, but haven’t really done anything beyond copying a few other files/scripts that I needed for some things.
To summarize my changes, most of them are just running yum updates so I don’t have old packages and then adding a few scripts. I tried:
VOLUME /srv
but either way this just gives me an empty volume at /mnt/myhostmount
1 Like
@matthewb just curious if you had an info here? This solution doesn’t seem to work for me unfortunately, see my above post.
1 Like
That’s probably why. You should not make any changes to our official images. Any modifications are not supported.
1 Like
Hmm ok, well here’s what I have added to the image:
- I added a step where we run
yum update -ysince some of the packages are a bit out of date. - I added a step where we add the aws cli package.
Beyond that we didn’t touch anything else in the image itself. If the above steps cause the bind mount to fail, then I’ll try the default image again.
1 Like
is the folder on the host empty (/mnt/myhostmount)? If so you’re describing expected behavior…
Normally the docker command docker create --volume /srv --name pmm-data percona/pmm-server:2 /bin/true would initialize the volume and copy the contents of /srv from the pmm-server image to the /srv directory in the volume…so when you later mount that volume the data needed is staged.
in your case I think you would first need to get a backup of the /srv directory of a regular pmm-server and restore that data to your bind-mount (and change the permissions as well).
Here are the backup instructions with the restore commands (mostly it’s the permission change stuff).
1 Like
I think the misconception for most is that when you initialize the pmm-server we initialize /srv but we don’t, it ships with the files and folders needed to run…we only run upgrades against it but not create if not existing.
1 Like
Thanks for clarifying @steve.hoffman That was indeed my assumption.
1 Like
Maybe I am misunderstanding here, and I am happy to make this into a new thread if needed but here’s what going on:
If I create a volume from the command, that works fine, that implies that docker is going to be managing the volume data - and the way our servers are built usually have a small root volume (maybe 30 GB where docker lives as well i.e. /var/lib/docker). And then a larger /mnt volume where data lives.
That being said, when I bind-mount i.e.:
docker run -v /mnt/myhostmount:/srv -p 443:443 percona/pmm-server:2
The container fails to run as nothing is copied to /mnt/myhostmount, and /srv in the container is empty.
/dev/nvme0n1p1 30G 16G 15G 51% /
/dev/mapper/myhostmount 1000G 33M 1000G 1% /mnt/myhostmount
docker run -it -v /mnt/myhostmount/percona-pmm:/srv -p 443:443 --entrypoint=/bin/bash --name pmm-test pmm-server
[root@b35b1418c520/ PerconaPmm server opt]# ls /srv/
[root@b35b1418c520/ PerconaPmm server opt]# cd /srv/
[root@b35b1418c520/ PerconaPmm server srv]# ls -las
total 0
0 drwxr-xr-x 2 root root 6 Feb 2 18:50 .
0 drwxr-xr-x 1 root root 6 Feb 2 18:51 ..
[root@b35b1418c520/ PerconaPmm server srv]#
total 0
0 drwxr-xr-x 2 root root 6 Feb 2 11:50 .
0 drwxr-xr-x 4 root root 41 Feb 2 11:50 ..```
1 Like
Right…what I’m saying is that there is no automatic process that creates all the stuff for /srv if it’s empty at start.
so if you ran something like:
docker run --detach --name pmm-server percona/pmm-server:2 for the sole purpose of getting the base content out of the /srv directory
docker cp pmm-server:/srv /tmp/
you can then stage that data in your /mnt/myhostmount
cp -af /tmp/srv/* /mnt/myhostmount
stop and delete that pmm-server
docker stop pmm-server && docker rm pmm-server
Now when you run your original command with the bind-mount
docker run -v /mnt/myhostmount:/srv -p 443:443 percona/pmm-server:2
you’ll at least have data in the /srv directory but it still won’t start because you’ll have permissions issues
so running
docker exec -it pmm-server chown -R root:root /srv && \
docker exec -it pmm-server chown -R pmm:pmm /srv/alertmanager && \
docker exec -it pmm-server chown -R root:pmm /srv/clickhouse && \
docker exec -it pmm-server chown -R grafana:grafana /srv/grafana && \
docker exec -it pmm-server chown -R pmm:pmm /srv/logs && \
docker exec -it pmm-server chown -R postgres:postgres /srv/postgres && \
docker exec -it pmm-server chown -R pmm:pmm /srv/prometheus && \
docker exec -it pmm-server chown -R pmm:pmm /srv/victoriametrics && \
docker exec -it pmm-server chown -R postgres:postgres /srv/logs/postgresql.log
should set the permissions back to what they need to be BUT what I don’t know is how the bind-mount reacts to changing ownership and preserving since these files live on the local filesystem and you may have to create users on the host system with the appropriate UID for pmm, grafana and postgres.
1 Like
Ah cool thanks, that makes a lot of sense, appreciate the reply.
Just wanted to clarify to make sure, this helps a lot at least on the setup, thanks again!
1 Like