PMM server - control over disk space used by pmm-server container

Percona Monitoring and Management (PMM) PMM 1.x
1

Greetings,

We use PMM 1.17 container-based Server to monitor MySQL DBs/hosts.
PMM Server has been installed in a Centos 7.5 container.
Currently that PMM Server is monitoring just one PMM Client host - with Centos 6.8 and MySQL 5.6.26 running there.

It seems that the PMM Server’s container is using more and more disk space.
We tried to limit the size of collected and kept data.

The server/container has been run with parameters of keeping history (retention) of 8 days and with 5 seconds pooling interval , as :
docker run -d
-p 80:80
–volumes-from pmm-data
–name pmm-server
–restart always
-e metrics_retention=192h -e queries_retention=8 -e metrics_resolution=5s
percona/pmm-server:1.17.0

While observing the PMM collected data - it seems that the QAN data indeed can be drilled-down over the last 8 days (while still presenting high-level graphs of 30 last days). However, the MySQL collected information (for example, “MySQL Overview”) can be drilled down for the last 30 days, and not only the 8 days as expected. So it seems that our retention parameters does not work as expected.

So, a few questions :

  1. What is the proper way to limit PMM collected information (namely QAN and MySQL oriented) to be kept for last N days, and not as set by the default ?
  2. Where/how is set/configured the total PMM Server’s container disk space storage ? Because it can be seen that after some threshold of 42G-43G has been reached - the disk storage stops to grow.
  3. Is it a proper way to define the “retention” parameters in the “docker run” command - or there is a better way ?

Best regards and looking forward your assistance,
Avi Vainshtein

2

Hi avi vainshtein
You are looking in the right places. Let me elaborate: [LIST=1]
[]metrics_retention is normally written METRICS_RETENTION, i’m not sure if entrypoint.sh correctly handles case sensitivity or not. But you can verify what 1.17 is actually using by visiting [url]https://your-server/prometheus/flags[/url] and check the value for storage.tsdb.retention
[]You can’t set a max limit using 1.17 for disk space usage, but I believe Prometheus team is working on this in a future release. You are correct though, the way Prometheus works is it collects metrics, stores them, and as it reaches the retention value it automatically purges the tail to make room for incoming fresh data. so if you were to maintain a static list of hosts/services (a fixed amount of metric series) you’ll see disk consumption grow until it hits retention level , and then should flatline.
[*]When we release PMM 2.0 we are building out ways to modify the server using GUI and API - but for right now the docker way is the recommended and supported way. What’s actually happening is that entrypoint.sh is receiving your environment variables (i.e. METRICS_RETENTION) and writing the prometheus startup flags (supervisord).
[/LIST] I hope this helps!

3

Thank you, @Michael Coburn, very much. Indeed, we used the settings in lowercase and may be that’s the reason of no impact.
Will try with uppercase and see the results.
regards