# Pmm graphs not visible **URL:** \ **Category:** PMM 1.x\ **Tags:** pmm\ **Created:** [April 6, 2020, 7:58am UTC](https://forums.percona.com/t/pmm-graphs-not-visible/7511 "2020-04-06T07:58:01Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![skumar](https://avatars.discourse-cdn.com/v4/letter/s/e9c0ed/32.png) [@skumar](https://forums.percona.com/u/skumar)\ **Post date:** [April 6, 2020, 7:58am UTC](https://forums.percona.com/t/pmm-graphs-not-visible/7511/1 "2020-04-06T07:58:01Z")
We have installed the latest pmm in one of the MySQL servers, everything was fine but not able to get the graphs. I have checked the pmm-admin list which shows its okay but the pmm-admin check-network shows the status DOWN. Here are the outputs, could you please me out.  > [09:13:44 root@prod-mysql-01.fra1 ~]$ pmm-admin list > pmm-admin 1.17.3 > > PMM Server      | [prod-perconamm-01.lga1.admarketplace.net](http://prod-perconamm-01.lga1.admarketplace.net/) (password-protected) > Client Name     | [prod-mysql-01.fra1.admarketplace.net](http://prod-mysql-01.fra1.admarketplace.net/) > Client Address  | 10.69.1.10 > Service Manager | linux-systemd > > -------------- ------------------------------------- ----------- -------- -------------------------------------------- --------------- > SERVICE TYPE   NAME                                  LOCAL PORT  RUNNING  DATA SOURCE                                  OPTIONS > -------------- ------------------------------------- ----------- -------- -------------------------------------------- --------------- > linux:metrics  [prod-mysql-01.fra1.admarketplace.net](http://prod-mysql-01.fra1.admarketplace.net/)  42000       YES      - > mysql:metrics  [prod-mysql-01.fra1.admarketplace.net](http://prod-mysql-01.fra1.admarketplace.net/)  42002       YES      pmm\_app:_@unix(/var/lib/mysql/mysql.sock)  tablestats=OFF > [09:13:50 root@prod-mysql-01.fra1 ~]$ pmm-admin check-network > PMM Network Status_ > > _Server Address | [prod-perconamm-01.lga1.admarketplace.net](http://prod-perconamm-01.lga1.admarketplace.net/) > Client Address | 10.69.1.10_ > > _System Time > NTP Server ([0.pool.ntp.org](http://0.pool.ntp.org/))         | unable to get ntp time: %!s(\) > PMM Server                          | 2020-04-06 13:14:02 +0000 GMT > PMM Client                          | 2020-04-06 09:14:16 -0400 EDT > PMM Client to PMM Server Time Drift | OK_ > > _Connection: Client –\> Server > -------------------- ------- > SERVER SERVICE       STATUS > -------------------- ------- > Consul API           OK > Prometheus API       OK > Query Analytics API  DOWN_ > > _Connection duration | 192.407236ms > Request duration    | -97.706805ms > Full round trip     | 94.700431ms_ > > > Connection: Client \<– Server > -------------- ------------------------------------- ----------------- ------- ---------- --------- > SERVICE TYPE   NAME                                  REMOTE ENDPOINT   STATUS  HTTPS/TLS  PASSWORD > -------------- ------------------------------------- ----------------- ------- ---------- --------- > linux:metrics  [prod-mysql-01.fra1.admarketplace.net](http://prod-mysql-01.fra1.admarketplace.net/)  [10.69.1.10:42000](http://10.69.1.10:42000/)  DOWN    YES        YES > mysql:metrics  [prod-mysql-01.fra1.admarketplace.net](http://prod-mysql-01.fra1.admarketplace.net/)  [10.69.1.10:42002](http://10.69.1.10:42002/)  DOWN    YES        YES > > When an endpoint is down it may indicate that the corresponding service is stopped (run ‘pmm-admin list’ to verify). > If it’s running, check out the logs /var/log/pmm-\*.log > > When all endpoints are down but ‘pmm-admin list’ shows they are up and no errors in the logs, > check the firewall settings whether this system allows incoming connections from server to address:port in question. > > Also you can check the endpoint status by the URL: [http://prod-perconamm-01.lga1.admarketplace.net/prometheus/targets](http://prod-perconamm-01.lga1.admarketplace.net/prometheus/targets) > > [09:14:17 root@prod-mysql-01.fra1 ~]$ ---
**Author:** ![steve.hoffman](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/steve.hoffman/32/66_2.png) [@steve.hoffman](https://forums.percona.com/u/steve.hoffman)\ **Post date:** [April 6, 2020, 8:52am UTC](https://forums.percona.com/t/pmm-graphs-not-visible/7511/2 "2020-04-06T08:52:06Z")
Is it possible the server you’re monitoring has multiple interfaces, one of which the PMM server cannot connect to for routing reasons (mgmt interface or something)?  When you register your pmm-client to the pmm-server we make a guess about which node-address is the right one to use and sometimes get it wrong so the PMM server tries to talk back to an interface that is not routable from server –\> client (would show up as errors here: [http://prod-perconamm-01.lga1.admarketplace.net/prometheus/targets](https://forums.percona.com/home/leaving?target=http%3A%2F%2Fprod-perconamm-01.lga1.admarketplace.net%2Fprometheus%2Ftargets "Link: https://forums.percona.com/home/leaving?target=http%3A%2F%2Fprod-perconamm-01.lga1.admarketplace.net%2Fprometheus%2Ftargets"), look for context exceeded messages in the Error column).  You may need to use the pmm-admin remove command to drop the node and re-add it with a different node-address defined.  It’s also possible that the firewall rules are preventing the PMM server from making a TCP connection on port 42000 and 42002 (you could check that from the PMM server with a simple telnet test to one of those two ports using 10.69.1.10 as the destination IP) ---
**Author:** ![skumar](https://avatars.discourse-cdn.com/v4/letter/s/e9c0ed/32.png) [@skumar](https://forums.percona.com/u/skumar)\ **Post date:** [April 6, 2020, 9:21am UTC](https://forums.percona.com/t/pmm-graphs-not-visible/7511/3 "2020-04-06T09:21:02Z")
I have checked the  [http://prod-perconamm-01.lga1.admarketplace.net/prometheus/targets](https://forums.percona.com/home/leaving?target=http%3A%2F%2Fprod-perconamm-01.lga1.admarketplace.net%2Fprometheus%2Ftargets), look for context exceeded messages in the Error column) and found the following error. ![](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/1X/4c0500787f5fcce0eae5c79940ea8eb0e725d04f.png) ---
**Author:** ![steve.hoffman](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/steve.hoffman/32/66_2.png) [@steve.hoffman](https://forums.percona.com/u/steve.hoffman)\ **Post date:** [April 6, 2020, 9:41am UTC](https://forums.percona.com/t/pmm-graphs-not-visible/7511/4 "2020-04-06T09:41:57Z")
Yes, that screams that the PMM server cannot talk to either the client node (routing - can the pmm server ping the ip of the client from a command prompt) or if the server can see the client’s IP, can you telnet first from the machine hosting the pmm server (‘telnet 10.69.1.10 42002’, if this fails there’s likely a firewall between pmm-server host and pmm-client host).  If all of that succeeds then we’d have to go inside the pmm-server container to run some tests and I’d shift my attention to selinux or something like that.  Has this worked for other systems in your network or is this the first system you’re attempting to monitor? ---
**Author:** ![skumar](https://avatars.discourse-cdn.com/v4/letter/s/e9c0ed/32.png) [@skumar](https://forums.percona.com/u/skumar)\ **Post date:** [April 6, 2020, 11:05am UTC](https://forums.percona.com/t/pmm-graphs-not-visible/7511/5 "2020-04-06T11:05:26Z")
We have opened the ports and its working now, thanks for the help.  ```auto [12:30:39 root@prod-mysql-01.fra1 ~]$ pmm-admin list pmm-admin 1.17.3 PMM Server | prod-perconamm-01.lga1.admarketplace.net (password-protected) Client Name | prod-mysql-01.fra1.admarketplace.net Client Address | 10.69.1.10 Service Manager | linux-systemd -------------- ------------------------------------- ----------- -------- -------------------------------------------- --------------- SERVICE TYPE NAME LOCAL PORT RUNNING DATA SOURCE OPTIONS -------------- ------------------------------------- ----------- -------- -------------------------------------------- --------------- linux:metrics prod-mysql-01.fra1.admarketplace.net 42000 YES - mysql:metrics prod-mysql-01.fra1.admarketplace.net 42002 YES pmm_app:***@unix(/var/lib/mysql/mysql.sock) tablestats=OFF [12:30:42 root@prod-mysql-01.fra1 ~]$ pmm-admin check-network PMM Network Status Server Address | prod-perconamm-01.lga1.admarketplace.net Client Address | 10.69.1.10 * System Time NTP Server (0.pool.ntp.org) | unable to get ntp time: %!s(<nil>) PMM Server | 2020-04-06 16:30:47 +0000 GMT PMM Client | 2020-04-06 12:31:02 -0400 EDT PMM Client to PMM Server Time Drift | OK * Connection: Client --> Server -------------------- ------- SERVER SERVICE STATUS -------------------- ------- Consul API OK Prometheus API OK Query Analytics API DOWN Connection duration | 195.003963ms Request duration | -99.663039ms Full round trip | 95.340924ms * Connection: Client <-- Server -------------- ------------------------------------- ----------------- ------- ---------- --------- SERVICE TYPE NAME REMOTE ENDPOINT STATUS HTTPS/TLS PASSWORD -------------- ------------------------------------- ----------------- ------- ---------- --------- linux:metrics prod-mysql-01.fra1.admarketplace.net 10.69.1.10:42000 OK YES YES mysql:metrics prod-mysql-01.fra1.admarketplace.net 10.69.1.10:42002 OK YES YES [12:31:03 root@prod-mysql-01.fra1 ~]$ ``` ---
**Author:** ![skumar](https://avatars.discourse-cdn.com/v4/letter/s/e9c0ed/32.png) [@skumar](https://forums.percona.com/u/skumar)\ **Post date:** [April 7, 2020, 7:13am UTC](https://forums.percona.com/t/pmm-graphs-not-visible/7511/6 "2020-04-07T07:13:16Z")
We still the following error after opening the ports. ![](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/1X/76c2b787ecb36452c66fbf7684675915779359f9.png "Image: /uploads/percona1/original/1X/76c2b787ecb36452c66fbf7684675915779359f9.png") ---
**Author:** ![steve.hoffman](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/steve.hoffman/32/66_2.png) [@steve.hoffman](https://forums.percona.com/u/steve.hoffman)\ **Post date:** [April 7, 2020, 8:37am UTC](https://forums.percona.com/t/pmm-graphs-not-visible/7511/7 "2020-04-07T08:37:15Z")
From the pmm server host (not inside the container) can you telnet to 10.69.1.10 on port 42002?  I’d at least like to verify that TCP isn’t part of the problem. Are there multiple interfaces on the 10.69.1.10 server and is that the IP that the PMM server should be talking to the PMM client on?  That error is either because you cannot talk to the port from the pmm server (network issue) or that pmm-managed isn’t running on the client end (maybe check and see if pmm-managed is running or restart pmm-agent)  ---
**Author:** ![skumar](https://avatars.discourse-cdn.com/v4/letter/s/e9c0ed/32.png) [@skumar](https://forums.percona.com/u/skumar)\ **Post date:** [April 8, 2020, 7:43am UTC](https://forums.percona.com/t/pmm-graphs-not-visible/7511/8 "2020-04-08T07:43:31Z")
we had some junk to be cleaned and reopen the ports which worked ---
**Author:** ![skumar](https://avatars.discourse-cdn.com/v4/letter/s/e9c0ed/32.png) [@skumar](https://forums.percona.com/u/skumar)\ **Post date:** [April 10, 2020, 9:34am UTC](https://forums.percona.com/t/pmm-graphs-not-visible/7511/9 "2020-04-10T09:34:25Z")
I have checked the  [http://prod-perconamm-01.lga1.admarketplace.net/prometheus/targets](https://forums.percona.com/home/leaving?target=http%3A%2F%2Fprod-perconamm-01.lga1.admarketplace.net%2Fprometheus%2Ftargets "Link: https://forums.percona.com/home/leaving?target=http%3A%2F%2Fprod-perconamm-01.lga1.admarketplace.net%2Fprometheus%2Ftargets"), look for context exceeded messages in the Error column) and found the following error. ![](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/1X/4c0500787f5fcce0eae5c79940ea8eb0e725d04f.png "Image: /uploads/percona1/original/1X/4c0500787f5fcce0eae5c79940ea8eb0e725d04f.png") ---
**Author:** ![skumar](https://avatars.discourse-cdn.com/v4/letter/s/e9c0ed/32.png) [@skumar](https://forums.percona.com/u/skumar)\ **Post date:** [April 10, 2020, 9:34am UTC](https://forums.percona.com/t/pmm-graphs-not-visible/7511/10 "2020-04-10T09:34:25Z")
I have checked the  [http://prod-perconamm-01.lga1.admarketplace.net/prometheus/targets](https://forums.percona.com/home/leaving?target=http%3A%2F%2Fprod-perconamm-01.lga1.admarketplace.net%2Fprometheus%2Ftargets "Link: https://forums.percona.com/home/leaving?target=http%3A%2F%2Fprod-perconamm-01.lga1.admarketplace.net%2Fprometheus%2Ftargets"), look for context exceeded messages in the Error column) and found the following error. ![](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/1X/4c0500787f5fcce0eae5c79940ea8eb0e725d04f.png "Image: /uploads/percona1/original/1X/4c0500787f5fcce0eae5c79940ea8eb0e725d04f.png")