Not the answer you need?
Register and ask your own question!

pmm-server was unable to connect pmm-client to collect linux:metrics

ivishnu7ivishnu7 ContributorCurrent User Role Advisor
pmm-server was unable to connect pmm-client to collect linux:metrics. The follow is the output of pmm-admin check-network. I checked pmm-admin list and linux:metrics showing as running. I verified firewall issue also. There is no firewall issue between pmm-server and pmm-client for 42000 port.

* Connection: Client <-- Server





SERVICE TYPE NAME REMOTE ENDPOINT STATUS HTTPS/TLS PASSWORD





linux:metrics pmm-client client_ip_address::42000 DOWN YES -
mysql:metrics dbcrpmysqlsbxha2 client_ip_address:42002 OK YES -









SERVICE TYPE NAME LOCAL PORT RUNNING DATA SOURCE OPTIONS





linux:metrics pmm_client ip address 42000 YES -


Thanks,
Vishnu

Comments

  • MykolaMykola Percona Percona Staff Role
    Hi Vishnu,

    lets check node_exporter.
    can you run the following command on pmm client itself and on pmm server?
    wget https://pmm-client-ip:42000/metrics --no-check-certificate
    
    I am wondering wget command output (not url content)
  • ivishnu7ivishnu7 Contributor Current User Role Advisor
    Thanks Mykola for your help!

    I tried the command (wget https://10.49.xx.xx:42000/metrics --no-check-certificate) from client and server and getting Connection refused error.

    Follow is the error message from pmm-server targets GUI (http://10.49.xx.xx/prometheus/targets)

    State: Down
    Error: context deadline exceeded


    Follow is the error from pmm-linux-metrics-42000.log.

    time="2017-03-29T12:09:44-07:00" level=info msg="Starting node_exporter (version=1.1.1, branch=master, revision=2d78e22000779d63c714011e4fb30c65623b9c77)" source="node_exporter.go:170"
    time="2017-03-29T12:09:44-07:00" level=info msg="Build context (go=go1.7.4, user=, date=)" source="node_exporter.go:171"
    time="2017-03-29T12:09:44-07:00" level=info msg="Enabled collectors:" source="node_exporter.go:190"
    time="2017-03-29T12:09:44-07:00" level=info msg=" - stat" source="node_exporter.go:192"
    time="2017-03-29T12:09:44-07:00" level=info msg=" - vmstat" source="node_exporter.go:192"
    time="2017-03-29T12:09:44-07:00" level=info msg=" - filesystem" source="node_exporter.go:192"
    time="2017-03-29T12:09:44-07:00" level=info msg=" - meminfo" source="node_exporter.go:192"
    time="2017-03-29T12:09:44-07:00" level=info msg=" - netdev" source="node_exporter.go:192"
    time="2017-03-29T12:09:44-07:00" level=info msg=" - netstat" source="node_exporter.go:192"
    time="2017-03-29T12:09:44-07:00" level=info msg=" - uname" source="node_exporter.go:192"
    time="2017-03-29T12:09:44-07:00" level=info msg=" - diskstats" source="node_exporter.go:192"
    time="2017-03-29T12:09:44-07:00" level=info msg=" - filefd" source="node_exporter.go:192"
    time="2017-03-29T12:09:44-07:00" level=info msg=" - loadavg" source="node_exporter.go:192"
    time="2017-03-29T12:09:44-07:00" level=info msg=" - time" source="node_exporter.go:192"
    time="2017-03-29T12:09:44-07:00" level=info msg="HTTPS/TLS is enabled" source="node_exporter.go:235"
    time="2017-03-29T12:09:44-07:00" level=info msg="Listening on 10.49.xx.xx:42000" source="node_exporter.go:238"
    2017/03/29 12:09:55 http: TLS handshake error from 10.49.xx.xx:44816: tls: first record does not look like a TLS handshake


    Thanks,
    Vishnu
  • MykolaMykola Percona Percona Staff Role
    So it looks like firewall or network configuration issue.
    prometheus (on PMM Server side) fetches https://pmm-client-ip:42000/metrics url every second.
    this url should be accessible from PMM Server.
    can you open 42000, 42002 ports for PMM Server ?
  • ivishnu7ivishnu7 Contributor Current User Role Advisor
    It doesn't seem firewall issue. Because I am able to do telnet from pmm-server to pmm-client with port number.

    [[email protected] log]# telnet dbcrpmysqlsbxha3 42000
    Trying 10.49.80.46...
    Connected to dbcrpmysqlsbxha2.

    [[email protected] log]# telnet dbcrpmysqlsbxha2 42002
    Trying 10.49.80.46...
    Connected to dbcrpmysqlsbxha2.

    What is context deadline exceeded error? and Where can I see error logs for more information?

    Thanks,
    Vishnu
  • MykolaMykola Percona Percona Staff Role
    Hm,

    is the following command working find on pmm client?
     wget https://pmm-client-ip:42000/metrics --no-check-certificate
    
  • ivishnu7ivishnu7 Contributor Current User Role Advisor
    Yes, It seems working fine. See the output of wget https://dbcrpmysqlsbxha2:42000/metrics --no-check-certificate command.

    [[email protected] ~]# wget https://dbcrpmysqlsbxha2:42000/metrics --no-check-certificate
    --2017-03-30 10:22:37-- https://dbcrpmysqlsbxha2:42000/metrics
    Resolving dbcrpmysqlsbxha2... 10.49.80.46
    Connecting to dbcrpmysqlsbxha2|10.49.80.46|:42000... connected.
    WARNING: cannot verify dbcrpmysqlsbxha2’s certificate, issued by “/O=PMM Client”:
    Unable to locally verify the issuer’s authority.
    WARNING: certificate common name “” doesn't match requested host name “dbcrpmysqlsbxha2”.
    HTTP request sent, awaiting response... 200 OK
    Length: 17784427 (17M) [text/plain]
    Saving to: “metrics”

    100%[===========================================================================================================================================================>] 17,784,427 6.35M/s in 2.7s

    2017-03-30 10:24:06 (6.35 MB/s) - “metrics” saved [17784427/17784427]




    Thanks,
    Vishnu
  • ivishnu7ivishnu7 Contributor Current User Role Advisor
    And additional to this, how to configure SMTP options to set mail notification alerts. I am using followed options and unable to get notifications. Please let me know where I need to modify below SMTP configuration.

    #################################### SMTP / Emailing ##########################
    [smtp]
    ;enabled = true
    ;host = localhost:25
    ;user = [email protected]
    ;password =
    ;cert_file =
    ;key_file =
    ;skip_verify = false
    ;from_address = [email protected]
  • MykolaMykola Percona Percona Staff Role
    about SMTP - I created separate topic https://www.percona.com/forums/questions-discussions/percona-monitoring-and-management/47988-configure-smtp

    about network connectivity,
    is the same wget command working fine on PMM Server machine?
  • ivishnu7ivishnu7 Contributor Current User Role Advisor
    Yes, wget https://dbcrpmysqlsbxha2:42000/metrics --no-check-certificate command working fine in pmm-server also.

    [[email protected] ~]# wget https://dbcrpmysqlsbxha2:42000/metrics --no-check-certificate
    --2017-03-31 10:10:09-- https://dbcrpmysqlsbxha2:42000/metrics
    Resolving dbcrpmysqlsbxha2... 10.49.80.46
    Connecting to dbcrpmysqlsbxha2|10.49.80.46|:42000... connected.
    WARNING: cannot verify dbcrpmysqlsbxha2’s certificate, issued by “/O=PMM Client”:
    Unable to locally verify the issuer’s authority.
    WARNING: certificate common name “” doesn't match requested host name “dbcrpmysqlsbxha2”.
    HTTP request sent, awaiting response... 200 OK
    Length: 17784140 (17M) [text/plain]
    Saving to: “metrics.1”

    100%[===========================================================================================================================================================>] 17,784,140 2.41M/s in 7.0s

    2017-03-31 10:11:37 (2.41 MB/s) - “metrics.1” saved [17784140/17784140]

    [[email protected] ~]#
    [[email protected] ~]#
    [[email protected] ~]# telnet dbcrpmysqlsbxha2 42000
    Trying 10.49.80.46...
    Connected to dbcrpmysqlsbxha2.
  • MykolaMykola Percona Percona Staff Role
    can you check https://dbcrpmysqlsbxha2:42000/metrics status on targets page?
    http://pmm-server-ip/prometheus/targets
    it should be "UP"
  • ivishnu7ivishnu7 Contributor Current User Role Advisor
    Thanks Mykola, The issue has resolved now. The problem is with interval timeout to connect pmm-server with pmm-client. Previously it was 1sec, Now I changed to 5 minutes. After that, everything seems fine.

    These are present my configuration variables in /etc.prometheus.yml:

    scrape_interval: 5s
    scrape_timeout: 5s
  • MykolaMykola Percona Percona Staff Role
    it is recommended to keep 1s because internal database performance debug requires high resolution.
    if you want to keep it is needed to found found what kind of monitoring query is slow and disable query via mysqld_exporter options.
    sometimes mysqld_exporter queries are ok, in this case it is needed place servers in one physical network or high performance network without delays.
  • ivishnu7ivishnu7 Contributor Current User Role Advisor
    Hi Mykola,

    How should I know the exact reasons for context deadline exceeded error? As you mentioned, I tried with scrape_interval 1s only. But I am seeing context deadlock error exceed error for some monitoring servers (Not for all servers) (Present I am monitoring 5 mysql instance from remote PMM server with 16 GB RAM and 4 CPU's. In this, I am able to collect linux and mysql metrics for 2 instances without any issue and not able to collect linux metrics for remaining 3 instances.). Getting context deadline exceeded errors in prometheus/target page. How to control linux metrics monitoring more efficiently? No firewall issue.
  • MykolaMykola Percona Percona Staff Role
    Hi Vishnu,

    `context deadline exceeded` means that mysqld_exporter works longer that prometheus expected (cannot finish work in 1 second).
    so mysqld_exporter creates long additional load on database server.
    mysqld_exporter runs many queries to database, so we can disable some checks to speedup it.
    usually most long query is 'tablestat', it is possible to disable this query by the following commands.
    pmm-admin remove mysql:metrics
    pmm-admin add mysql:metrics --disable-tablestats
    
    also options available.
  • ivishnu7ivishnu7 Contributor Current User Role Advisor
    Thanks Mykola for update. Here you mentioned all checks only for mysql (--disable-tablestats, --disable-userstats --disable-processlist --disable-binlogstats). We already disabled table-stats in my environment to improve performance and everything seems good. Is there any way to filter linux metrics like mysql metrics? Because I am seeing context deadline exceeded errors only for Linux metrics.

    Thanks PMM team for providing beautiful monitoring tool for MySQL.

    Thnaks,
    Vishnu
  • MykolaMykola Percona Percona Staff Role
    usually, node_expoerter (Linux metrics) works fine. we don't have any options for it.

    can you share the output of the following command?
    time curl -s -o /dev/null --insecure  https://pmm-client-ip:42000/metrics
    
    please run it on pmm-client and on pmm-server (I want to compare response time)
  • Anita2020Anita2020 Current User Role Participant

    Hello,

    if pmm-server was unable to connect pmm - client to collect linux: metrics, Please go through this link:https://www.percona.com/doc/percona-monitoring-and-management/2.x/manage/client-linux-metrics.html

    Thanks,

    Anita

  • RohitRohit Current User Role Participant
    I also faced same type of issue which might be due to some latency between client node and server node. Same has been resolved by increasing value of scrape_timeout to 5s .
  • RohitRohit Current User Role Participant
    I also faced same type of issue which might be due to some latency between client node and server node. Same has been resolved by increasing value of scrape_timeout to 5s .
Sign In or Register to comment.

MySQL, InnoDB, MariaDB and MongoDB are trademarks of their respective owners.
Copyright ©2005 - 2020 Percona LLC. All rights reserved.