Pmm-agent and insane memory usage

We’re testing PMM 2.12 on a dedicated database server with 300GB memory. When the pmm-agent service is enabled, the mysql process slowly grows in memory usage (approx. ~2GB/ day) until it uses up all of the memory on the server and forces us to restart mysql to free up the memory.

I notice on https://www.percona.com/doc/percona-monitoring-and-management/2.x/setting-up/client/mysql.html it says,

MySQL with too many tables can lead to PMM Server overload due to the streaming of too much time series data. It can also lead to too many queries from mysqld_exporter causing extra load on MySQL. Therefore PMM Server disables most consuming mysqld_exporter collectors automatically if there are more than 1000 tables.

Above it claims to automatically deal with this, but if this isn’t what we’re running in to, I wonder if it’s not something similar?

We have gone through the above linked pages and feel we have sufficiently followed directions in setting up our servers for use with Percona, but are also open to suggestions or answering questions in helping to debug this.

Thank you.

A little more information:

in our setup, pmm-agent and mysql run on a dedicated DB server, while pmm-server and docker are on a separate utility instance. So the above quoted mention may not be relevant to our problem, after all? It seems pmm-agent and all of the scraping of mysql might be more relevant to the cause? By the way, we have it set to scrape every 10 seconds. Thanks again.

Ok, just to understand correctly:

You have a DB server with 300GB of RAM running MySQL and you’ve installed pmm-client 2.12.0 (“client”) on it, you have a separate system running pmm-server 2.12.0 in the docker container (“server”). Over time you see your MySQL process consume roughly 2GB of RAM per day on the client before needing to be restarted in order to gain the memory back.

Assuming I’ve got this much correct are you looking for help understanding what’s consuming the memory in the MySQL process or are you thinking that the PMM instance is causing your MySQL instance to swell and eventually run out of memory?

If you have over 1000 tables on your mysql system that can both impact performance on your client AND the server because of the load polling for table level stats in your mysql instance particularly for a very active database as well as on the server trying to crunch all the data. For this reason we automatically disable table stats when you add the mysql client to monitoring (pmm-admin add mysql …) and you should have seen a message along the lines of: Table statistics collection enabled (the limit is 1000, the actual table count is 602). But I don’t think that would lead to a memory issue on your client particularly with the MySQL process but CPU and Disk would go way up…I guess I could think of a few scenarios where RAM would go up but that’d be a configuration issue and would show up regardless of pmm-client being installed.

I guess I’d have to ask…if pmm-client weren’t installed were you also seeing memory increases or is this only observed after installing the client?

You have a DB server with 300GB of RAM running MySQL and you’ve installed pmm-client 2.12.0 (“client”) on it, you have a separate system running pmm-server 2.12.0 in the docker container (“server”). Over time you see your MySQL process consume roughly 2GB of RAM per day on the client before needing to be restarted in order to gain the memory back.

All Correct.

Assuming I’ve got this much correct are you looking for help understanding what’s consuming the memory in the MySQL process or are you thinking that the PMM instance is causing your MySQL instance to swell and eventually run out of memory?

if pmm-client weren’t installed were you also seeing memory increases or is this only observed after installing the client?

This is correct, we only run out of memory when pmm-agent is enabled and running. If I stop pmm-agent, the problem goes away. With pmm-agent enabled, it’s only a matter of time before MySQL will begin to use 250GB+ of memory, if we don’t intervene the server will basically crash!

You have a DB server with 300GB of RAM running MySQL and you’ve installed pmm-client 2.12.0 (“client”) on it, you have a separate system running pmm-server 2.12.0 in the docker container (“server”).

This is all correct.

Over time you see your MySQL process consume roughly 2GB of RAM per day on the client before needing to be restarted in order to gain the memory back.

if pmm-client weren’t installed were you also seeing memory increases or is this only observed after installing the client?

This is only observed when the client is running. if we stop pmm-agent, the problem goes away. If we leave it running, mysql eventually uses upwards of 250GB+ RAM and needs an intervention.

Well, I’m going to go out on a limb and assume that with that much ram you’re not running a small wordpress db but if you’re under 1000 tables, we could still see a tablestats impact for massively large tables. I’d say the first and easiest thing to try is to re-add the mysql monitoring with the --disable-tablestats flag. That will eliminate at lest one item we know can have an impact. From there, here’s a good blog with some memory debugging tips that can help narrow down what’s consuming the memory which we can then see which ones PMM would have a direct impact on and I’ll post this topic in our internal channels to see if any of our many performance tuning experts have addition advice on what could be the issue as you’re able to find more info.