# Client Doesnt show up in Grafana

**URL:** <https://forums.percona.com/t/client-doesnt-show-up-in-grafana/5321>\
**Category:** PMM 1.x\
**Created:** [December 26, 2016, 11:50am UTC](https://forums.percona.com/t/client-doesnt-show-up-in-grafana/5321 "2016-12-26T11:50:34Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![kooshals](https://avatars.discourse-cdn.com/v4/letter/k/ea666f/32.png) [@kooshals](https://forums.percona.com/u/kooshals)\
**Post date:** [December 26, 2016, 11:50am UTC](https://forums.percona.com/t/client-doesnt-show-up-in-grafana/5321/1 "2016-12-26T11:50:34Z")

</div>

My client doesnt show up in Grafana. Pmm server is in aws machine and it has ports 80 and 443 open.  
Does it need any more ports open for incoming connections from pmm client?  
[root@XYZ sbin]# pmm-admin ping  
OK, PMM server is alive.

PMM Server | XX.YY.ZZ.AA  
Client Name | ABC  
Client Address | AA.BB.CC.DD

---

<div class="post-metadata">

**Author:** ![Mykola](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/mykola/32/30_2.png) [@Mykola](https://forums.percona.com/u/Mykola)\
**Post date:** [December 27, 2016, 5:03am UTC](https://forums.percona.com/t/client-doesnt-show-up-in-grafana/5321/2 "2016-12-27T05:03:39Z")

</div>

It is needed to allow traffic:  
[LIST]  
[_]from PMM client any port to PMM server 80 and 443 ports  
[_]from PMM server any port to PMM client port 42000 for linux:metrics  
[_]from PMM server any port to PMM client port 42002 for mysql:metrics  
[_]from PMM server any port to PMM client port 42003 for mongodb:metrics  
[\*]from PMM server any port to PMM client port 42004 for proxysql:metrics  
[/LIST]

---

<div class="post-metadata">

**Author:** ![lux](https://avatars.discourse-cdn.com/v4/letter/l/8c91f0/32.png) [@lux](https://forums.percona.com/u/lux)\
**Post date:** [January 10, 2017, 4:54am UTC](https://forums.percona.com/t/client-doesnt-show-up-in-grafana/5321/3 "2017-01-10T04:54:53Z")

</div>

Hi,  
I am having the same problem. I am pretty sure they (clients and the server) are in the same network and the traffic is allowed, have checked the ports are opened with nmap.  
I can see the servers in the Query Analytics but they are not shown in Metrics Monitor. I am having trouble with just some clients, most works fine.

pmm-admin 1.0.7  
Go Version 1.7.4

---

<div class="post-metadata">

**Author:** ![Mykola](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/mykola/32/30_2.png) [@Mykola](https://forums.percona.com/u/Mykola)\
**Post date:** [January 10, 2017, 5:30am UTC](https://forums.percona.com/t/client-doesnt-show-up-in-grafana/5321/4 "2017-01-10T05:30:23Z")

</div>

Hi lux,

you can check connectivity by ‘pmm-admin check-network’ command (run it on problem host).  
it checks connectivity from server to client and from client to server.

---

<div class="post-metadata">

**Author:** ![lux](https://avatars.discourse-cdn.com/v4/letter/l/8c91f0/32.png) [@lux](https://forums.percona.com/u/lux)\
**Post date:** [January 10, 2017, 7:04am UTC](https://forums.percona.com/t/client-doesnt-show-up-in-grafana/5321/5 "2017-01-10T07:04:57Z")

</div>

Hi,  
Thanks for quick reply. check-network shows status is down, and i found these in prometheus.log from server. Do you know what causes it?

time=“2017-01-10T12:23:57Z” level=warning msg=“Scrape health sample discarded” error=“sample timestamp out of order” sample=up{instance=“client”, job=“mysql”} =\> 0 @[1484051037.723] source=“scrape.go:500”  
time=“2017-01-10T12:23:57Z” level=warning msg=“Scrape duration sample discarded” error=“sample timestamp out of order” sample=scrape\_duration\_seconds{instance=“client”, job=“mysql”} =\> 0.256849543 @[1484051037.723] source=“scrape.go:503”

i checked the timestamp on the client, it is correct.

---

<div class="post-metadata">

**Author:** ![lux](https://avatars.discourse-cdn.com/v4/letter/l/8c91f0/32.png) [@lux](https://forums.percona.com/u/lux)\
**Post date:** [January 10, 2017, 7:07am UTC](https://forums.percona.com/t/client-doesnt-show-up-in-grafana/5321/6 "2017-01-10T07:07:51Z")

</div>

This is the message from [url][http://server/prometheus/targets[/url]](http://server/prometheus/targets%5B/url%5D)

Get [url][http://ip:42000/metrics:[/url]](http://ip:42000/metrics:%5B/url%5D) malformed HTTP response “\x15\x03\x01\x00\x02\x02”

---

<div class="post-metadata">

**Author:** ![Mykola](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/mykola/32/30_2.png) [@Mykola](https://forums.percona.com/u/Mykola)\
**Post date:** [January 10, 2017, 7:38am UTC](https://forums.percona.com/t/client-doesnt-show-up-in-grafana/5321/7 "2017-01-10T07:38:38Z")

</div>

can you check, may be you have two clients with the same name/hostname?

---

<div class="post-metadata">

**Author:** ![lux](https://avatars.discourse-cdn.com/v4/letter/l/8c91f0/32.png) [@lux](https://forums.percona.com/u/lux)\
**Post date:** [January 10, 2017, 9:14am UTC](https://forums.percona.com/t/client-doesnt-show-up-in-grafana/5321/8 "2017-01-10T09:14:25Z")

</div>

well, I checked [URL][http://server/consul/[/URL]](http://server/consul/%5B/URL%5D), there is no redundant server/host name, prometheus/targets shows IP address which is individual and nothing identical either. I have tried to deregister the client from consul and did “pmm-admin repair” from the client, still the same problem. Do I miss some required packages or something?  
It is bit strange that I only have a couple clients which don’t work, all others work fine. And they are all in the same network, so shouldn’t have different firewall rules between them.

---

<div class="post-metadata">

**Author:** ![lux](https://avatars.discourse-cdn.com/v4/letter/l/8c91f0/32.png) [@lux](https://forums.percona.com/u/lux)\
**Post date:** [January 10, 2017, 9:21am UTC](https://forums.percona.com/t/client-doesnt-show-up-in-grafana/5321/9 "2017-01-10T09:21:56Z")

</div>

1. I checked all pmm-\* logs, there is no error/complain other than “tls: first record does not look like a TLS handshake”, which seems common/normal from others reply.
2. pmm-admin check-network shows client to server works fine, but server to client status is down.
3. I used nmap from server to client on port 42000 and 42002, both shows open.

I am bit running out of ideas what else to check…

☹

---

<div class="post-metadata">

**Author:** ![Mykola](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/mykola/32/30_2.png) [@Mykola](https://forums.percona.com/u/Mykola)\
**Post date:** [January 10, 2017, 10:10am UTC](https://forums.percona.com/t/client-doesnt-show-up-in-grafana/5321/10 "2017-01-10T10:10:56Z")

</div>

can you check on broken client:

- Are versions of PMM server and PMM client the same?
- Can you show output of ‘pmm-admin check-network’ command? (you can replace IPs by 1.1.1.1 and 2.2.2.2)

---

<div class="post-metadata">

**Author:** ![lux](https://avatars.discourse-cdn.com/v4/letter/l/8c91f0/32.png) [@lux](https://forums.percona.com/u/lux)\
**Post date:** [January 11, 2017, 4:15am UTC](https://forums.percona.com/t/client-doesnt-show-up-in-grafana/5321/11 "2017-01-11T04:15:37Z")

</div>

Thanks for the tips. Yeah finally it works.

The server version was 1.0.5 and the client which didn’t work was 1.0.7. After upgrading server to 1.0.7, i can see all clients now. I checked the client which worked with server 1.0.5, they were having 1.0.6, so didn’t think the version matters that much. Anyway, thanks for helping! 🙂

---

<div class="post-metadata">

**Author:** ![TerranUp16](https://avatars.discourse-cdn.com/v4/letter/t/b2d939/32.png) [@TerranUp16](https://forums.percona.com/u/TerranUp16)\
**Post date:** [January 31, 2017, 3:35pm UTC](https://forums.percona.com/t/client-doesnt-show-up-in-grafana/5321/12 "2017-01-31T15:35:50Z")

</div>

I am having a similar issue, except both my client and server instances are running the same version.

```auto
pmm-admin check-network
PMM Network Status

Server Address | REDACTED
Client Address | REDACTED

* System Time
Server | 2017-01-31 15:57:20 -0500 EST
Client | 2017-01-31 15:57:20 -0500 EST
Time Drift | OK

* Connection: Client --> Server
-------------------- -------
SERVER SERVICE STATUS
-------------------- -------
Consul API OK
Prometheus API OK
Query Analytics API OK

Connection duration | 616.433µs
Request duration | 1.107953ms
Full round trip | 1.724386ms

* Connection: Client <-- Server
-------------- -------------------- ---------------------- ------- ---------- ---------
SERVICE TYPE NAME REMOTE ENDPOINT STATUS HTTPS/TLS PASSWORD
-------------- -------------------- ---------------------- ------- ---------- ---------
linux:metrics REDACTED REDACTED:42000 DOWN YES YES
mysql:metrics REDACTED REDACTED:42002 DOWN YES YES

When an endpoint is down it may indicate that the corresponding service is stopped (run 'pmm-admin list' to verify).
If it's running, check out the logs /var/log/pmm-*.log

When all endpoints are down but 'pmm-admin list' shows they are up and no errors in the logs,
check the firewall settings whether this system allows incoming connections from server to address:port in question.

Also you can check the endpoint status by the URL: http://REDACTED/prometheus/targets

```

```auto
pmm-admin 1.0.7

PMM Server | REDACTED (password-protected)
Client Name | REDACTED
Client Address | REDACTED
Service Manager | linux-systemd

-------------- -------------------- ----------- -------- ---------------------------------------- ---------------------
SERVICE TYPE NAME LOCAL PORT RUNNING DATA SOURCE OPTIONS
-------------- -------------------- ----------- -------- ---------------------------------------- ---------------------
mysql:queries REDACTED - YES pmm:***&#64;unix(/var/lib/mysql/mysql.sock) query_source=slowlog
linux:metrics REDACTED 42000 YES -
mysql:metrics REDACTED 42002 YES pmm:***&#64;unix(/var/lib/mysql/mysql.sock) tablestats=OFF

```

I ran cURL commands from within the Docker container to the endpoints without issue-

```auto
root&#64;2105dab46d05:/opt# curl --insecure https://REDACTED:42000
<html>
<head><title>Node Exporter</title></head>
<body>
<h1>Node Exporter</h1>
<p><a href="/metrics">Metrics</a></p>
</body>
</html>root&#64;2105dab46d05:/opt

```

For the client-side logs:

/var/log/pmm-mysql-queries-0.log

```auto
2017/01/31 16:04:08.182228 WARNING data-sender Waiting for API to ack qan_1485892380005193039: read tcp REDACTED:48005->REDACTED:80: i/o timeout

```

The other two logs just have the TLS handshake errors.

Are there any further avenues of exploration?

---

<div class="post-metadata">

**Author:** ![Mykola](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/mykola/32/30_2.png) [@Mykola](https://forums.percona.com/u/Mykola)\
**Post date:** [February 1, 2017, 1:02am UTC](https://forums.percona.com/t/client-doesnt-show-up-in-grafana/5321/13 "2017-02-01T01:02:06Z")

</div>

Hi TerranUp16,

Do you have two networks (like private and public)?
