# pmm-server was unable to connect pmm-client to collect linux:metrics
**URL:** https://forums.percona.com/t/pmm-server-was-unable-to-connect-pmm-client-to-collect-linux-metrics/5513
**Category:** PMM 1.x
**Created:** [March 28, 2017, 5:58pm UTC](https://forums.percona.com/t/pmm-server-was-unable-to-connect-pmm-client-to-collect-linux-metrics/5513 "2017-03-28T17:58:08Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@ivishnu7](https://forums.percona.com/u/ivishnu7)
#### Post date: [March 28, 2017, 5:58pm UTC](https://forums.percona.com/t/pmm-server-was-unable-to-connect-pmm-client-to-collect-linux-metrics/5513/1 "2017-03-28T17:58:08Z")
pmm-server was unable to connect pmm-client to collect linux:metrics. The follow is the output of pmm-admin check-network. I checked pmm-admin list and linux:metrics showing as running. I verified firewall issue also. There is no firewall issue between pmm-server and pmm-client for 42000 port.
- Connection: Client ← Server
* * *
SERVICE TYPE NAME REMOTE ENDPOINT STATUS HTTPS/TLS PASSWORD
* * *
linux:metrics pmm-client client\_ip\_address::42000 DOWN YES -
mysql:metrics dbcrpmysqlsbxha2 client\_ip\_address:42002 OK YES -
* * *
SERVICE TYPE NAME LOCAL PORT RUNNING DATA SOURCE OPTIONS
* * *
linux:metrics pmm\_client ip address 42000 YES -
Thanks,
Vishnu
---
### Author:  [@Mykola](https://forums.percona.com/u/Mykola)
#### Post date: [March 29, 2017, 12:17am UTC](https://forums.percona.com/t/pmm-server-was-unable-to-connect-pmm-client-to-collect-linux-metrics/5513/2 "2017-03-29T00:17:48Z")
Hi Vishnu,
lets check [node\_exporter](https://github.com/prometheus/node_exporter).
can you run the following command on pmm client itself and on pmm server?
```auto
wget https://pmm-client-ip:42000/metrics --no-check-certificate
```
I am wondering wget command output (not url content)
---
### Author:  [@ivishnu7](https://forums.percona.com/u/ivishnu7)
#### Post date: [March 29, 2017, 1:47pm UTC](https://forums.percona.com/t/pmm-server-was-unable-to-connect-pmm-client-to-collect-linux-metrics/5513/3 "2017-03-29T13:47:05Z")
Thanks Mykola for your help!
I tried the command (wget [url][https://10.49.xx.xx:42000/metrics[/url]](https://10.49.xx.xx:42000/metrics%5B/url%5D) --no-check-certificate) from client and server and getting Connection refused error.
Follow is the error message from pmm-server targets GUI ([url][http://10.49.xx.xx/prometheus/targets[/url]](http://10.49.xx.xx/prometheus/targets%5B/url%5D))
State: Down
Error: context deadline exceeded
Follow is the error from pmm-linux-metrics-42000.log.
time=“2017-03-29T12:09:44-07:00” level=info msg=“Starting node\_exporter (version=1.1.1, branch=master, revision=2d78e22000779d63c714011e4fb30c65623b9c77)” source=“node\_exporter.go:170”
time=“2017-03-29T12:09:44-07:00” level=info msg=“Build context (go=go1.7.4, user=, date=)” source=“node\_exporter.go:171”
time=“2017-03-29T12:09:44-07:00” level=info msg=“Enabled collectors:” source=“node\_exporter.go:190”
time=“2017-03-29T12:09:44-07:00” level=info msg=" - stat" source=“node\_exporter.go:192”
time=“2017-03-29T12:09:44-07:00” level=info msg=" - vmstat" source=“node\_exporter.go:192”
time=“2017-03-29T12:09:44-07:00” level=info msg=" - filesystem" source=“node\_exporter.go:192”
time=“2017-03-29T12:09:44-07:00” level=info msg=" - meminfo" source=“node\_exporter.go:192”
time=“2017-03-29T12:09:44-07:00” level=info msg=" - netdev" source=“node\_exporter.go:192”
time=“2017-03-29T12:09:44-07:00” level=info msg=" - netstat" source=“node\_exporter.go:192”
time=“2017-03-29T12:09:44-07:00” level=info msg=" - uname" source=“node\_exporter.go:192”
time=“2017-03-29T12:09:44-07:00” level=info msg=" - diskstats" source=“node\_exporter.go:192”
time=“2017-03-29T12:09:44-07:00” level=info msg=" - filefd" source=“node\_exporter.go:192”
time=“2017-03-29T12:09:44-07:00” level=info msg=" - loadavg" source=“node\_exporter.go:192”
time=“2017-03-29T12:09:44-07:00” level=info msg=" - time" source=“node\_exporter.go:192”
time=“2017-03-29T12:09:44-07:00” level=info msg=“HTTPS/TLS is enabled” source=“node\_exporter.go:235”
time=“2017-03-29T12:09:44-07:00” level=info msg=“Listening on 10.49.xx.xx:42000” source=“node\_exporter.go:238”
2017/03/29 12:09:55 http: TLS handshake error from 10.49.xx.xx:44816: tls: first record does not look like a TLS handshake
Thanks,
Vishnu
---
### Author:  [@Mykola](https://forums.percona.com/u/Mykola)
#### Post date: [March 30, 2017, 12:32am UTC](https://forums.percona.com/t/pmm-server-was-unable-to-connect-pmm-client-to-collect-linux-metrics/5513/4 "2017-03-30T00:32:14Z")
So it looks like firewall or network configuration issue.
prometheus (on PMM Server side) fetches [https://pmm-](https://pmm-)client-ip:42000/metrics url every second.
this url should be accessible from PMM Server.
can you open 42000, 42002 ports for PMM Server ?
---
### Author:  [@ivishnu7](https://forums.percona.com/u/ivishnu7)
#### Post date: [March 30, 2017, 1:13am UTC](https://forums.percona.com/t/pmm-server-was-unable-to-connect-pmm-client-to-collect-linux-metrics/5513/5 "2017-03-30T01:13:38Z")
It doesn’t seem firewall issue. Because I am able to do telnet from pmm-server to pmm-client with port number.
[root@dbcrpmysqlsbxha3 log]# telnet dbcrpmysqlsbxha3 42000
Trying 10.49.80.46…
Connected to dbcrpmysqlsbxha2.
[root@dbcrpmysqlsbxha3 log]# telnet dbcrpmysqlsbxha2 42002
Trying 10.49.80.46…
Connected to dbcrpmysqlsbxha2.
What is context deadline exceeded error? and Where can I see error logs for more information?
Thanks,
Vishnu
---
### Author:  [@Mykola](https://forums.percona.com/u/Mykola)
#### Post date: [March 30, 2017, 1:55am UTC](https://forums.percona.com/t/pmm-server-was-unable-to-connect-pmm-client-to-collect-linux-metrics/5513/6 "2017-03-30T01:55:23Z")
Hm,
is the following command working find on pmm client?
```auto
wget https://pmm-client-ip:42000/metrics --no-check-certificate
```
---
### Author:  [@ivishnu7](https://forums.percona.com/u/ivishnu7)
#### Post date: [March 30, 2017, 12:13pm UTC](https://forums.percona.com/t/pmm-server-was-unable-to-connect-pmm-client-to-collect-linux-metrics/5513/7 "2017-03-30T12:13:56Z")
Yes, It seems working fine. See the output of wget [url][https://dbcrpmysqlsbxha2:42000/metrics[/url]](https://dbcrpmysqlsbxha2:42000/metrics%5B/url%5D) --no-check-certificate command.
[root@dbcrpmysqlsbxha2 ~]# wget [url][https://dbcrpmysqlsbxha2:42000/metrics[/url]](https://dbcrpmysqlsbxha2:42000/metrics%5B/url%5D) --no-check-certificate
–2017-03-30 10:22:37-- [url][https://dbcrpmysqlsbxha2:42000/metrics[/url]](https://dbcrpmysqlsbxha2:42000/metrics%5B/url%5D)
Resolving dbcrpmysqlsbxha2… 10.49.80.46
Connecting to dbcrpmysqlsbxha2|10.49.80.46|:42000… connected.
WARNING: cannot verify dbcrpmysqlsbxha2’s certificate, issued by “/O=PMM Client”:
Unable to locally verify the issuer’s authority.
WARNING: certificate common name “” doesn’t match requested host name “dbcrpmysqlsbxha2”.
HTTP request sent, awaiting response… 200 OK
Length: 17784427 (17M) [text/plain]
Saving to: “metrics”
100%[===========================================================================================================================================================\>] 17,784,427 6.35M/s in 2.7s
2017-03-30 10:24:06 (6.35 MB/s) - “metrics” saved [17784427/17784427]
Thanks,
Vishnu
---
### Author:  [@ivishnu7](https://forums.percona.com/u/ivishnu7)
#### Post date: [March 30, 2017, 4:33pm UTC](https://forums.percona.com/t/pmm-server-was-unable-to-connect-pmm-client-to-collect-linux-metrics/5513/8 "2017-03-30T16:33:54Z")
And additional to this, how to configure SMTP options to set mail notification alerts. I am using followed options and unable to get notifications. Please let me know where I need to modify below SMTP configuration.
#################################### SMTP / Emailing ##########################
[smtp]
;enabled = true
;host = localhost:25
;user = [ivishnu7@gmail.com](mailto:ivishnu7@gmail.com)
;password =
;cert\_file =
;key\_file =
;skip\_verify = false
;from\_address = [admin@grafana.localhost](mailto:admin@grafana.localhost)
---
### Author:  [@Mykola](https://forums.percona.com/u/Mykola)
#### Post date: [March 31, 2017, 12:06am UTC](https://forums.percona.com/t/pmm-server-was-unable-to-connect-pmm-client-to-collect-linux-metrics/5513/9 "2017-03-31T00:06:51Z")
about SMTP - I created separate topic [url][https://www.percona.com/forums/questions-discussions/percona-monitoring-and-management/47988-configure-smtp[/url]](https://www.percona.com/forums/questions-discussions/percona-monitoring-and-management/47988-configure-smtp%5B/url%5D)
about network connectivity,
is the same wget command working fine on PMM Server machine?
---
### Author:  [@ivishnu7](https://forums.percona.com/u/ivishnu7)
#### Post date: [March 31, 2017, 11:44am UTC](https://forums.percona.com/t/pmm-server-was-unable-to-connect-pmm-client-to-collect-linux-metrics/5513/10 "2017-03-31T11:44:40Z")
Yes, wget [url][https://dbcrpmysqlsbxha2:42000/metrics[/url]](https://dbcrpmysqlsbxha2:42000/metrics%5B/url%5D) --no-check-certificate command working fine in pmm-server also.
[root@dbcrpmysqlsbxha3 ~]# wget [url][https://dbcrpmysqlsbxha2:42000/metrics[/url]](https://dbcrpmysqlsbxha2:42000/metrics%5B/url%5D) --no-check-certificate
–2017-03-31 10:10:09-- [url][https://dbcrpmysqlsbxha2:42000/metrics[/url]](https://dbcrpmysqlsbxha2:42000/metrics%5B/url%5D)
Resolving dbcrpmysqlsbxha2… 10.49.80.46
Connecting to dbcrpmysqlsbxha2|10.49.80.46|:42000… connected.
WARNING: cannot verify dbcrpmysqlsbxha2’s certificate, issued by “/O=PMM Client”:
Unable to locally verify the issuer’s authority.
WARNING: certificate common name “” doesn’t match requested host name “dbcrpmysqlsbxha2”.
HTTP request sent, awaiting response… 200 OK
Length: 17784140 (17M) [text/plain]
Saving to: “metrics.1”
100%[===========================================================================================================================================================\>] 17,784,140 2.41M/s in 7.0s
2017-03-31 10:11:37 (2.41 MB/s) - “metrics.1” saved [17784140/17784140]
[root@dbcrpmysqlsbxha3 ~]#
[root@dbcrpmysqlsbxha3 ~]#
[root@dbcrpmysqlsbxha3 ~]# telnet dbcrpmysqlsbxha2 42000
Trying 10.49.80.46…
Connected to dbcrpmysqlsbxha2.
---
### Author:  [@Mykola](https://forums.percona.com/u/Mykola)
#### Post date: [April 4, 2017, 12:43am UTC](https://forums.percona.com/t/pmm-server-was-unable-to-connect-pmm-client-to-collect-linux-metrics/5513/11 "2017-04-04T00:43:04Z")
can you check [url][https://dbcrpmysqlsbxha2:42000/metrics[/url]](https://dbcrpmysqlsbxha2:42000/metrics%5B/url%5D) status on targets page?
[url][http://pmm-server-ip/prometheus/targets[/url]](http://pmm-server-ip/prometheus/targets%5B/url%5D)
it should be “UP”
---
### Author:  [@ivishnu7](https://forums.percona.com/u/ivishnu7)
#### Post date: [April 10, 2017, 6:06pm UTC](https://forums.percona.com/t/pmm-server-was-unable-to-connect-pmm-client-to-collect-linux-metrics/5513/12 "2017-04-10T18:06:46Z")
Thanks Mykola, The issue has resolved now. The problem is with interval timeout to connect pmm-server with pmm-client. Previously it was 1sec, Now I changed to 5 minutes. After that, everything seems fine.
These are present my configuration variables in /etc.prometheus.yml:
scrape\_interval: 5s
scrape\_timeout: 5s
---
### Author:  [@Mykola](https://forums.percona.com/u/Mykola)
#### Post date: [April 11, 2017, 12:31am UTC](https://forums.percona.com/t/pmm-server-was-unable-to-connect-pmm-client-to-collect-linux-metrics/5513/13 "2017-04-11T00:31:33Z")
it is recommended to keep 1s because internal database performance debug requires high resolution.
if you want to keep it is needed to found found what kind of monitoring query is slow and disable query via mysqld\_exporter options.
sometimes mysqld\_exporter queries are ok, in this case it is needed place servers in one physical network or high performance network without delays.
---
### Author:  [@ivishnu7](https://forums.percona.com/u/ivishnu7)
#### Post date: [May 16, 2017, 9:08pm UTC](https://forums.percona.com/t/pmm-server-was-unable-to-connect-pmm-client-to-collect-linux-metrics/5513/14 "2017-05-16T21:08:20Z")
Hi Mykola,
How should I know the exact reasons for context deadline exceeded error? As you mentioned, I tried with scrape\_interval 1s only. But I am seeing context deadlock error exceed error for some monitoring servers (Not for all servers) (Present I am monitoring 5 mysql instance from remote PMM server with 16 GB RAM and 4 CPU’s. In this, I am able to collect linux and mysql metrics for 2 instances without any issue and not able to collect linux metrics for remaining 3 instances.). Getting context deadline exceeded errors in prometheus/target page. How to control linux metrics monitoring more efficiently? No firewall issue.
---
### Author:  [@Mykola](https://forums.percona.com/u/Mykola)
#### Post date: [May 26, 2017, 12:16am UTC](https://forums.percona.com/t/pmm-server-was-unable-to-connect-pmm-client-to-collect-linux-metrics/5513/15 "2017-05-26T00:16:05Z")
Hi Vishnu,
`context deadline exceeded` means that mysqld\_exporter works longer that prometheus expected (cannot finish work in 1 second).
so mysqld\_exporter creates long additional load on database server.
mysqld\_exporter runs many queries to database, so we can disable some checks to speedup it.
usually most long query is ‘tablestat’, it is possible to disable this query by the following commands.
```auto
pmm-admin remove mysql:metrics
pmm-admin add mysql:metrics --disable-tablestats
```
also options available.
---
### Author:  [@ivishnu7](https://forums.percona.com/u/ivishnu7)
#### Post date: [May 30, 2017, 1:05pm UTC](https://forums.percona.com/t/pmm-server-was-unable-to-connect-pmm-client-to-collect-linux-metrics/5513/16 "2017-05-30T13:05:46Z")
Thanks Mykola for update. Here you mentioned all checks only for mysql (–disable-tablestats, --disable-userstats --disable-processlist --disable-binlogstats). We already disabled table-stats in my environment to improve performance and everything seems good. Is there any way to filter linux metrics like mysql metrics? Because I am seeing context deadline exceeded errors only for Linux metrics.
Thanks PMM team for providing beautiful monitoring tool for MySQL.
Thnaks,
Vishnu
---
### Author:  [@Mykola](https://forums.percona.com/u/Mykola)
#### Post date: [May 31, 2017, 6:43am UTC](https://forums.percona.com/t/pmm-server-was-unable-to-connect-pmm-client-to-collect-linux-metrics/5513/17 "2017-05-31T06:43:02Z")
usually, node\_expoerter (Linux metrics) works fine. we don’t have any options for it.
can you share the output of the following command?
```auto
time curl -s -o /dev/null --insecure https://pmm-client-ip:42000/metrics
```
please run it on pmm-client and on pmm-server (I want to compare response time)
---
### Author:  [@Anita2020](https://forums.percona.com/u/Anita2020)
#### Post date: [September 7, 2020, 3:32pm UTC](https://forums.percona.com/t/pmm-server-was-unable-to-connect-pmm-client-to-collect-linux-metrics/5513/18 "2020-09-07T15:32:06Z")
Hello,
if pmm-server was unable to connect pmm - client to collect [linux](https://mindmajix.com/linux-training): metrics, Please go through this link:[https://www.percona.com/doc/percona-monitoring-and-management/2.x/manage/client-linux-metrics.html](https://www.percona.com/doc/percona-monitoring-and-management/2.x/manage/client-linux-metrics.html)
Thanks,
Anita
---
### Author:  [@ROHIT\_KHURANA](https://forums.percona.com/u/ROHIT_KHURANA)
#### Post date: [September 8, 2020, 8:10am UTC](https://forums.percona.com/t/pmm-server-was-unable-to-connect-pmm-client-to-collect-linux-metrics/5513/19 "2020-09-08T08:10:18Z")
I also faced same type of issue which might be due to some latency between client node and server node. Same has been resolved by increasing value of scrape\_timeout to 5s .
---
### Author:  [@ROHIT\_KHURANA](https://forums.percona.com/u/ROHIT_KHURANA)
#### Post date: [September 8, 2020, 8:10am UTC](https://forums.percona.com/t/pmm-server-was-unable-to-connect-pmm-client-to-collect-linux-metrics/5513/20 "2020-09-08T08:10:39Z")
I also faced same type of issue which might be due to some latency between client node and server node. Same has been resolved by increasing value of scrape\_timeout to 5s .
[Next page](https://forums.percona.com/t/pmm-server-was-unable-to-connect-pmm-client-to-collect-linux-metrics/5513.md?page=2)