What is the maximum number of mysql instances a pmm client can handle?

Percona Monitoring and Management (PMM) PMM 2.x
1

Hi.

I added a mysql instance (“NEW” from here on) to a pmm client. The configuration was accepted and “pmm-admin list” shows the NEW connection just like the previous ones (this particular client already has 9 mysql instances configured). However, the pmm server never showed any data of NEW.

The client and mysql Server are on the same network, so there is no firewalling involved here.

While debugging this, I removed other configurations from the client and noticed, that data from NEW would now also show up.

So I re-added old instances and apparently, the 10th and further instances never work (ie data is never shown on pmm server dashboards).

Is there a limit of 10 connections per pmm client or might this be a bug?

Thanks
Dominik

3 Likes
2

Hi @Dominik_Klein , welcome to the Percona forums!
This is an interesting question that I haven’t seen posed before. Do you have anything interesting in the logs?

journalctl -xe --unit=pmm-agent

also can you show the output of:

systemctl status pmm-agent

You should see one mysqld_exporter per monitored instance. Let me know if you see less than this.

1 Like
3

Hey @Michael_Coburn,

I collected the requested information but the forum does not let me upload since I am a new user. You can view it here: pmm client log · GitHub

But apart from that: I saw that all mysqld_exporter processes disappear once I configure the tenth instance.

So the initial report has to be somewhat clarified. The pmm-client stops sending any mysql data once the tenth instance is configured. As soon as you remove the tenth instance, all other instances start working again (mysqld_exporter processes are started). It does not matter which of the 10 instances is removed. The problem is not in the new database.

The client setup is as follows:
Debian 11.3
pmm2-client package in version 2.27.0-6.bullseye

The complete setup consists of 2 pmm machines (client and server) and several mysql instances that are all configured in the one client.

+------------+                +------------+
|            |                |            |
| pmm server |<---------------+ pmm client |
|            |                |            |
+------------+      +---------+--+---------+----+
                    |            |              |
                    |            |              |
                    |            |              |
                    |            |              |
                    |            |              |
                    |            |              |
                +---v-+      +---v-+         +--v---+
                |     |      |     |         |      |
                | db1 |      | db2 |   ...   | dbXY |
                |     |      |     |         |      |
                +-----+      +-----+         +------+
1 Like
4

I scanned through the logs and nothing jumped out other than one or two of the collectors timed out but seemed to be fine on the next run…

It’s a long shot since it’s unlikely someone changed it but what is the value you have for ports:min|max in your clients /usr/local/percona/pmm2/config/pmm-agent.yaml. Default is from 42000-51999. I’m not sure what would happen if you had the width of the port range to narrow for your usage. I’d imagine the Nth and beyond exporter wouldn’t be able to start but everything leading up to that would be just fine…so probably not your issue but worth a shot to check.

1 Like
5 
ports:
    min: 42000
    max: 51999
1 Like