PMM2 linux metrics only

Hi,

first a big thank you to the percona stuff. I love PMM!

Now to my problem/question. According to this https://www.percona.com/doc/percona-monitoring-and-management/2.x/manage/client-linux-metrics.html I just have to configure pmm-admin (I’ve uninstalled the old client and installed the pmm2-client package) to have linux metrics in the GUI. I did that with a few hosts where I only need linux metrics and where’s no mysql or mongodb running, but they don’t show up in the GUI. Did I miss something?

best regards
Oliver

In PMM2 you do not need to add Linux specifically. When you register host with PMM Server it will automatically start System (node) level monitoring.

If it does not something is wrong :slight_smile:

hm, ok, then there’s something wrong :wink:

Where can I start debugging? pmm-admin says the status is fine:

Agent ID: /agent_id/d6b75d41-3b68-4eb4-840f-6dc45f188e2d
Node ID : /node_id/9427049b-47e0-43de-a44d-db7d336c5baa

PMM Server:
URL : https://admin:xxx@10.20.30.2:8191/
Version: 2.0.0

PMM-agent:
Connected : true
Time drift: 233.771µs
Latency : 609.813µs

Agents:
/agent_id/4e4cc46d-559e-4c06-871b-ba87937e8673 NODE_EXPORTER RUNNING


pmm-admin list:

Service type Service name Address and port Service ID

Agent type Status Agent ID Service ID
pmm-agent connected /agent_id/d6b75d41-3b68-4eb4-840f-6dc45f188e2d
node_exporter running /agent_id/4e4cc46d-559e-4c06-871b-ba87937e8673


Using tcpdump I can see traffic between the clients and the pmm server. However they don’t show up in the GUI.
I’ve explored the swagger API and bit and saw this “/v1/inventory/Nodes/List” which successfully returns all configured clients.

Very strange… Can you go on your http://PMMSERVER/prometheus and enter “up” metric and click execute. Do you see the hosts which you only Monitor as Linux hosts out there ?

magenbrot , can you run pmm-admin summary it will provide more information.
Also, can you share Node Overview dashboard screenshot

hm, it seems like I can’t send you a direct message by this forum. I can’t see the form (I don’t want to post the zip file here, because it might contain private information).

But here’s the screenshot of the node overview.

Hi Peter,

in the up metric I can see all the hosts I’ve connected with pmm-admin:

prometheus-up-metric.txt (9.89 KB)

roma.novikov can you tell me how I can send you a private message? It seems that this function is not available for me.

magenbrotIt looks like a region was specified for only one of your node (tyr.ovtec.it). The latest grafana version hasn’t support empty values. So only nodes with regions are shown. Such behavior will be changed in next PMM releases. For now you should remove and add the node without a region or specify a region for other nodes by the same procedure.

Hi adivinho! Thank you for your response. Can you tell me how I can add hosts without a region?

Hi magenbrot , Please use the next command to remove/add your node tyr.ovtec.it

pmm-admin config --force tyr.ovtec.it

e.g.

# pmm-admin config --force ip-10-178-1-195
Checking local pmm-agent status...
pmm-agent is running.
Registering pmm-agent on PMM Server...
Registered.
Configuration file /usr/local/percona/pmm2/config/pmm-agent.yaml updated.
Reloading pmm-agent configuration...
Configuration reloaded.
Checking local pmm-agent status...
pmm-agent is running.
#

Can you make sure that port 42001 is open on the host being monitored? This worked for me.

Hi adivinho, I’ve done that for all my hosts, but I still can only see one host. I’ve even reset the pmm-server and pmm-data container to try it with a fresh setup.

I’ve used the following commands to start the container and to configure the clients (Ports 80 and 443 are not available on the pmm-server host, I’m using a reverse proxy to access the grafana GUI).

pmm-server:


docker create -v /srv --name pmm-data percona/pmm-server:2 /bin/true
docker run -d -p 10.20.30.2:8190:80 -p 10.20.30.2:8191:443 --volumes-from pmm-data --name pmm-server --restart always percona/pmm-server:2

pmm-client:


pmm-admin config --server-insecure-tls --server-url=https://admin:xxx@10.20.30.2:8191 --force tyr.ovtec.it

Thx, but I’ve no firewall on the internal interfaces which I use for pmm. I also had tcpdump running for about an hour on this port, but there where no packets.

Hi!

I finally found the problem with my setup. All my hosts except tyr.ovtec.it have two network interfaces, one with a public ip address and another one for the internal network. So the “usual” pmm-admin config added the hosts with their public IP address, but since this ip is firewalled the hosts did not appear in PMM.

I now added them with this command and everything is working fine:
pmm-admin config --server-insecure-tls --server-url=https://admin:password@10.20.30.2:8191 --force 10.20.30.6 generic loki.mydomain.de

in this example 10.20.30.2 is the host where the pmm is running in a docker container and 10.20.30.6 is the internal IP address of the host which I want to monitor.

Thanks again for all your help!

regards
Oli

Thank you for updating this post, it could help others in the future and we appreciate the share.

I seem that I have a similar problem (pmm2).

I don’t see any nginx metrics.

Hi @mosad,

Have you added any service for monitoring or configured pmm-agent on a node only?

pmm-agent only

It could be a firewall issue.

You may switch metrics flow mode to push in the latest PMM version 2.12.0

At first you should remove already added node on the inventory page.

Then please run configuration command again but with flag –metrics-mode=push

error: unknown long flag ‘–metrics-mode’