Pmm graphs not visible

skumar · December 30, 2020, 4:26pm

We have installed the latest pmm in one of the MySQL servers, everything was fine but not able to get the graphs.
I have checked the pmm-admin list which shows its okay but the pmm-admin check-network shows the status DOWN.

Here are the outputs, could you please me out.

[09:13:44 root@prod-mysql-01.fra1 ~]$ pmm-admin list
pmm-admin 1.17.3

PMM Server | prod-perconamm-01.lga1.admarketplace.net (password-protected)
Client Name | prod-mysql-01.fra1.admarketplace.net
Client Address | 10.69.1.10
Service Manager | linux-systemd

-------------- ------------------------------------- ----------- -------- -------------------------------------------- ---------------
SERVICE TYPE NAME LOCAL PORT RUNNING DATA SOURCE OPTIONS
-------------- ------------------------------------- ----------- -------- -------------------------------------------- ---------------
linux:metrics prod-mysql-01.fra1.admarketplace.net  42000 YES -
mysql:metrics prod-mysql-01.fra1.admarketplace.net  42002 YES pmm_app:@unix(/var/lib/mysql/mysql.sock) tablestats=OFF
[09:13:50 root@prod-mysql-01.fra1 ~]$ pmm-admin check-network
PMM Network Status

Server Address | prod-perconamm-01.lga1.admarketplace.net
Client Address | 10.69.1.10

System Time
NTP Server (0.pool.ntp.org) | unable to get ntp time: %!s(<nil>)
PMM Server | 2020-04-06 13:14:02 +0000 GMT
PMM Client | 2020-04-06 09:14:16 -0400 EDT
PMM Client to PMM Server Time Drift | OK

Connection: Client --> Server
-------------------- -------
SERVER SERVICE STATUS
-------------------- -------
Consul API OK
Prometheus API OK
Query Analytics API DOWN

Connection duration | 192.407236ms
Request duration | -97.706805ms
Full round trip | 94.700431ms

Connection: Client <-- Server
-------------- ------------------------------------- ----------------- ------- ---------- ---------
SERVICE TYPE NAME REMOTE ENDPOINT STATUS HTTPS/TLS PASSWORD
-------------- ------------------------------------- ----------------- ------- ---------- ---------
linux:metrics prod-mysql-01.fra1.admarketplace.net  10.69.1.10:42000  DOWN YES YES
mysql:metrics prod-mysql-01.fra1.admarketplace.net  10.69.1.10:42002  DOWN YES YES

When an endpoint is down it may indicate that the corresponding service is stopped (run ‘pmm-admin list’ to verify).
If it’s running, check out the logs /var/log/pmm-*.log

When all endpoints are down but ‘pmm-admin list’ shows they are up and no errors in the logs,
check the firewall settings whether this system allows incoming connections from server to address:port in question.

Also you can check the endpoint status by the URL: http://prod-perconamm-01.lga1.admarketplace.net/prometheus/targets

[09:14:17 root@prod-mysql-01.fra1 ~]$

steve.hoffman · December 30, 2020, 4:36pm

Is it possible the server you’re monitoring has multiple interfaces, one of which the PMM server cannot connect to for routing reasons (mgmt interface or something)? When you register your pmm-client to the pmm-server we make a guess about which node-address is the right one to use and sometimes get it wrong so the PMM server tries to talk back to an interface that is not routable from server --> client (would show up as errors here: http://prod-perconamm-01.lga1.admarketplace.net/prometheus/targets, look for context exceeded messages in the Error column). You may need to use the pmm-admin remove command to drop the node and re-add it with a different node-address defined. It’s also possible that the firewall rules are preventing the PMM server from making a TCP connection on port 42000 and 42002 (you could check that from the PMM server with a simple telnet test to one of those two ports using 10.69.1.10 as the destination IP)

skumar · December 30, 2020, 4:40pm

I have checked the http://prod-perconamm-01.lga1.admarketplace.net/prometheus/targets, look for context exceeded messages in the Error column) and found the following error.

steve.hoffman · December 30, 2020, 4:36pm

Yes, that screams that the PMM server cannot talk to either the client node (routing - can the pmm server ping the ip of the client from a command prompt) or if the server can see the client’s IP, can you telnet first from the machine hosting the pmm server (‘telnet 10.69.1.10 42002’, if this fails there’s likely a firewall between pmm-server host and pmm-client host). If all of that succeeds then we’d have to go inside the pmm-server container to run some tests and I’d shift my attention to selinux or something like that. Has this worked for other systems in your network or is this the first system you’re attempting to monitor?

skumar · December 30, 2020, 4:36pm

We have opened the ports and its working now, thanks for the help.

[12:30:39 root@prod-mysql-01.fra1 ~]$ pmm-admin list
pmm-admin 1.17.3

PMM Server | prod-perconamm-01.lga1.admarketplace.net (password-protected)
Client Name | prod-mysql-01.fra1.admarketplace.net
Client Address | 10.69.1.10
Service Manager | linux-systemd

-------------- ------------------------------------- ----------- -------- -------------------------------------------- ---------------
SERVICE TYPE NAME LOCAL PORT RUNNING DATA SOURCE OPTIONS
-------------- ------------------------------------- ----------- -------- -------------------------------------------- ---------------
linux:metrics prod-mysql-01.fra1.admarketplace.net 42000 YES -
mysql:metrics prod-mysql-01.fra1.admarketplace.net 42002 YES pmm_app:***@unix(/var/lib/mysql/mysql.sock) tablestats=OFF
[12:30:42 root@prod-mysql-01.fra1 ~]$ pmm-admin check-network
PMM Network Status

Server Address | prod-perconamm-01.lga1.admarketplace.net
Client Address | 10.69.1.10

* System Time
NTP Server (0.pool.ntp.org) | unable to get ntp time: %!s(&lt;nil&gt;)
PMM Server | 2020-04-06 16:30:47 +0000 GMT
PMM Client | 2020-04-06 12:31:02 -0400 EDT
PMM Client to PMM Server Time Drift | OK

* Connection: Client --&gt; Server
-------------------- -------
SERVER SERVICE STATUS
-------------------- -------
Consul API OK
Prometheus API OK
Query Analytics API DOWN

Connection duration | 195.003963ms
Request duration | -99.663039ms
Full round trip | 95.340924ms


* Connection: Client &lt;-- Server
-------------- ------------------------------------- ----------------- ------- ---------- ---------
SERVICE TYPE NAME REMOTE ENDPOINT STATUS HTTPS/TLS PASSWORD
-------------- ------------------------------------- ----------------- ------- ---------- ---------
linux:metrics prod-mysql-01.fra1.admarketplace.net 10.69.1.10:42000 OK YES YES
mysql:metrics prod-mysql-01.fra1.admarketplace.net 10.69.1.10:42002 OK YES YES

[12:31:03 root@prod-mysql-01.fra1 ~]$

skumar · December 30, 2020, 4:40pm

We still the following error after opening the ports.

steve.hoffman · December 30, 2020, 4:36pm

From the pmm server host (not inside the container) can you telnet to 10.69.1.10 on port 42002? I’d at least like to verify that TCP isn’t part of the problem. Are there multiple interfaces on the 10.69.1.10 server and is that the IP that the PMM server should be talking to the PMM client on? That error is either because you cannot talk to the port from the pmm server (network issue) or that pmm-managed isn’t running on the client end (maybe check and see if pmm-managed is running or restart pmm-agent)

skumar · December 30, 2020, 4:36pm

we had some junk to be cleaned and reopen the ports which worked

skumar · December 30, 2020, 4:40pm

I have checked the http://prod-perconamm-01.lga1.admarketplace.net/prometheus/targets, look for context exceeded messages in the Error column) and found the following error.

skumar · December 30, 2020, 4:40pm

I have checked the http://prod-perconamm-01.lga1.admarketplace.net/prometheus/targets, look for context exceeded messages in the Error column) and found the following error.