pmm-server not collecting mysql metrics from client

Percona Monitoring and Management (PMM) PMM 1.x
1

Hello,

I have installed pmm-server on RHEL 7.4 through docker and pmm-clients in two RHEL 6.6 servers.
I am able to collect OS metrics from clients, but not able to collect mysql metrics from clients.
I have telnet from the pmm-server to clients and from clients to pmm server.

pmm-admin list output
01:11:20 # pmm-admin list
pmm-admin 1.11.0

PMM Server | [server_ip]
Client Name | [client]
Client Address | [client_ip]
Service Manager | linux-upstart

SERVICE TYPE NAME LOCAL PORT RUNNING DATA SOURCE OPTIONS

mysql:queries [server] - YES root:@tcp(127.0.0.1:3306) query_source=slowlog, query_examples=true, slow_log_rotation=true, retain_slow_logs=1
linux:metrics [server] 42000 YES -
mysql:metrics [server] 42002 YES root:@tcp(127.0.0.1:3306) tablestats=OFF

pmm-admin check-network output

01:14:24 # pmm-admin check-network
PMM Network Status

Server Address | [server]:80
Client Address | [server]

  • System Time
    PMM Server | 2018-06-11 06:16:23 +0000 GMT
    PMM Client | 2018-06-11 01:16:29 -0500 CDT
    PMM Client to PMM Server Time Drift | OK

  • Connection: Client → Server

SERVER SERVICE STATUS

Consul API OK
Prometheus API OK
Query Analytics API OK

Connection duration | 712.884µs
Request duration | 6.964205ms
Full round trip | 7.677089ms

  • Connection: Client ← Server

SERVICE TYPE NAME REMOTE ENDPOINT STATUS HTTPS/TLS PASSWORD

linux:metrics [server] [server]:42000 OK YES -
mysql:metrics [server] [server]:42002 DOWN YES -

When an endpoint is down it may indicate that the corresponding service is stopped (run ‘pmm-admin list’ to verify).
If it’s running, check out the logs /var/log/pmm-*.log

When all endpoints are down but ‘pmm-admin list’ shows they are up and no errors in the logs,
check the firewall settings whether this system allows incoming connections from server to address:port in question.

Also you can check the endpoint status by the URL: http://[server]/prometheus/targets

output of /var/log/pmm-mysql-metrics-42002.log

time=“2018-06-08T11:56:02-05:00” level=info msg=“Starting mysqld_exporter (version=, branch=, revision=)” source=“mysqld_exporter.go:292”
time=“2018-06-08T11:56:02-05:00” level=info msg=“Build context (go=go1.9.4, user=, date=)” source=“mysqld_exporter.go:293”
time=“2018-06-08T11:56:02-05:00” level=info msg=“HTTPS/TLS is enabled” source=“mysqld_exporter.go:362”
time=“2018-06-08T11:56:02-05:00” level=info msg=“Enabled High Resolution scrapers:” source=“mysqld_exporter.go:376”
time=“2018-06-08T11:56:02-05:00” level=info msg=" --collect.info_schema.innodb_metrics" source=“mysqld_exporter.go:378”
time=“2018-06-08T11:56:02-05:00” level=info msg=" --collect.global_status" source=“mysqld_exporter.go:378”
time=“2018-06-08T11:56:02-05:00” level=info msg=“Enabled Medium Resolution scrapers:” source=“mysqld_exporter.go:382”
time=“2018-06-08T11:56:02-05:00” level=info msg=" --collect.info_schema.processlist" source=“mysqld_exporter.go:384”
time=“2018-06-08T11:56:02-05:00” level=info msg=" --collect.slave_status" source=“mysqld_exporter.go:384”
time=“2018-06-08T11:56:02-05:00” level=info msg=" --collect.perf_schema.eventswaits" source=“mysqld_exporter.go:384”
time=“2018-06-08T11:56:02-05:00” level=info msg=" --collect.perf_schema.file_events" source=“mysqld_exporter.go:384”
time=“2018-06-08T11:56:02-05:00” level=info msg=" --collect.info_schema.query_response_time" source=“mysqld_exporter.go:384”
time=“2018-06-08T11:56:02-05:00” level=info msg=“Enabled Low Resolution scrapers:” source=“mysqld_exporter.go:388”
time=“2018-06-08T11:56:02-05:00” level=info msg=" --collect.global_variables" source=“mysqld_exporter.go:390”
time=“2018-06-08T11:56:02-05:00” level=info msg=" --collect.binlog_size" source=“mysqld_exporter.go:390”
time=“2018-06-08T11:56:02-05:00” level=info msg=" --collect.info_schema.userstats" source=“mysqld_exporter.go:390”
time=“2018-06-08T11:56:02-05:00” level=info msg=“Listening on [server]:42002” source=“mysqld_exporter.go:399”
time=“2018-06-08T11:56:33-05:00” level=info msg=“Starting mysqld_exporter (version=, branch=, revision=)” source=“mysqld_exporter.go:292”
time=“2018-06-08T11:56:33-05:00” level=info msg=“Build context (go=go1.9.4, user=, date=)” source=“mysqld_exporter.go:293”
time=“2018-06-08T11:56:33-05:00” level=info msg=“HTTPS/TLS is enabled” source=“mysqld_exporter.go:362”
time=“2018-06-08T11:56:33-05:00” level=info msg=“Enabled High Resolution scrapers:” source=“mysqld_exporter.go:376”
time=“2018-06-08T11:56:33-05:00” level=info msg=" --collect.info_schema.innodb_metrics" source=“mysqld_exporter.go:378”
time=“2018-06-08T11:56:33-05:00” level=info msg=" --collect.global_status" source=“mysqld_exporter.go:378”
time=“2018-06-08T11:56:33-05:00” level=info msg=“Enabled Medium Resolution scrapers:” source=“mysqld_exporter.go:382”
time=“2018-06-08T11:56:33-05:00” level=info msg=" --collect.info_schema.processlist" source=“mysqld_exporter.go:384”
time=“2018-06-08T11:56:33-05:00” level=info msg=" --collect.slave_status" source=“mysqld_exporter.go:384”
time=“2018-06-08T11:56:33-05:00” level=info msg=" --collect.perf_schema.eventswaits" source=“mysqld_exporter.go:384”
time=“2018-06-08T11:56:33-05:00” level=info msg=" --collect.perf_schema.file_events" source=“mysqld_exporter.go:384”
time=“2018-06-08T11:56:33-05:00” level=info msg=" --collect.info_schema.query_response_time" source=“mysqld_exporter.go:384”
time=“2018-06-08T11:56:33-05:00” level=info msg=“Enabled Low Resolution scrapers:” source=“mysqld_exporter.go:388”
time=“2018-06-08T11:56:33-05:00” level=info msg=" --collect.global_variables" source=“mysqld_exporter.go:390”
time=“2018-06-08T11:56:33-05:00” level=info msg=" --collect.binlog_size" source=“mysqld_exporter.go:390”
time=“2018-06-08T11:56:33-05:00” level=info msg=" --collect.info_schema.userstats" source=“mysqld_exporter.go:390”
time=“2018-06-08T11:56:33-05:00” level=info msg=“Listening on [server]:42002” source=“mysqld_exporter.go:399”
2018/06/08 11:58:19 http: TLS handshake error from [server]:52196: tls: first record does not look like a TLS handshake
2018/06/08 12:20:33 http: Accept error: accept tcp [server]:42002: accept4: too many open files; retrying in 5ms
2018/06/08 12:20:33 http: Accept error: accept tcp [server]:42002: accept4: too many open files; retrying in 10ms
2018/06/08 12:20:33 http: Accept error: accept tcp [server]:42002: accept4: too many open files; retrying in 20ms
2018/06/08 12:20:33 http: Accept error: accept tcp [server]:42002: accept4: too many open files; retrying in 40ms
2018/06/08 12:20:33 http: Accept error: accept tcp [server]:42002: accept4: too many open files; retrying in 80ms
2018/06/08 12:20:33 http: Accept error: accept tcp [server]:42002: accept4: too many open files; retrying in 160ms
2018/06/08 12:20:33 http: Accept error: accept tcp [server]:42002: accept4: too many open files; retrying in 320ms

In prometheus dashboard for mysql metrics status is down.
Telnet is working fine and OS metrics is working, only mysql metrics it is giving this error.

Please help me in solving this.

2

curl command is working from docker container to pmm-client
docker exec -ti pmm-server bash
curl --insecure [URL=“https://10.2.21.xx:42002/”]https://xx.xx.xx.xx:42002

MySQLd 3-in-1 exporter

MySQL 3-in-1 exporter

  • high-res metrics
  • medium-res metrics
  • low-res metrics
    • [/URL]

    [url]http://xx.xx.xx.xx/prometheus/targets[/url] is giving down status for mysql metrics with context deadline exceeded.

    3

    This issue is fixed after rebooting the client server, don’t know the exact reason why it was failing to collect mysql metrics.

    Thanks

    4

    Thank you for letting us know, Jagannath it could help other users if they’re in a position to try a reboot. Appreciated.