Just installed an instance of PMM as a trial, monitoring 3 Percona XtraDB Cluster nodes.
After just 48 hours of statistics gathered, the Prometheus data storage has reached 9.7GiB. So, ~35GiB of storage to monitor just 3 MySQL servers for a week?
It’s 1s resolution by default and maximum statistics if all metrics are available.
Also there could be other factors.
For example, depending from number of tables - it can be a lot, and disabling per table stats (re-adding mysql:metrics with --disable-tablestats flag) will decrease disk usage dramatically.
Yes, you need to re-create pmm-server container to pass other env variables, you can preserve the data one…
Also there is an option to change chunk encoding for metric storage which claims 50% saving in cost of 20% CPU overhead.
However, we have not made any tests yet but will do soon.