Not the answer you need?
Register and ask your own question!

pmm-server not collecting mysql metrics from client

JagannathJagannath EntrantInactive User Role Beginner
Hello,

I have installed pmm-server on RHEL 7.4 through docker and pmm-clients in two RHEL 6.6 servers.
I am able to collect OS metrics from clients, but not able to collect mysql metrics from clients.
I have telnet from the pmm-server to clients and from clients to pmm server.

pmm-admin list output
01:11:20 # pmm-admin list
pmm-admin 1.11.0

PMM Server | [server_ip]
Client Name | [client]
Client Address | [client_ip]
Service Manager | linux-upstart






SERVICE TYPE NAME LOCAL PORT RUNNING DATA SOURCE OPTIONS





mysql:queries [server] - YES root:***@tcp(127.0.0.1:3306) query_source=slowlog, query_examples=true, slow_log_rotation=true, retain_slow_logs=1
linux:metrics [server] 42000 YES -
mysql:metrics [server] 42002 YES root:***@tcp(127.0.0.1:3306) tablestats=OFF

pmm-admin check-network output

01:14:24 # pmm-admin check-network
PMM Network Status

Server Address | [server]:80
Client Address | [server]

* System Time
PMM Server | 2018-06-11 06:16:23 +0000 GMT
PMM Client | 2018-06-11 01:16:29 -0500 CDT
PMM Client to PMM Server Time Drift | OK

* Connection: Client --> Server

SERVER SERVICE STATUS

Consul API OK
Prometheus API OK
Query Analytics API OK

Connection duration | 712.884µs
Request duration | 6.964205ms
Full round trip | 7.677089ms


* Connection: Client <-- Server





SERVICE TYPE NAME REMOTE ENDPOINT STATUS HTTPS/TLS PASSWORD





linux:metrics [server] [server]:42000 OK YES -
mysql:metrics [server] [server]:42002 DOWN YES -

When an endpoint is down it may indicate that the corresponding service is stopped (run 'pmm-admin list' to verify).
If it's running, check out the logs /var/log/pmm-*.log

When all endpoints are down but 'pmm-admin list' shows they are up and no errors in the logs,
check the firewall settings whether this system allows incoming connections from server to address:port in question.

Also you can check the endpoint status by the URL: http://[server]/prometheus/targets


output of /var/log/pmm-mysql-metrics-42002.log

time="2018-06-08T11:56:02-05:00" level=info msg="Starting mysqld_exporter (version=, branch=, revision=)" source="mysqld_exporter.go:292"
time="2018-06-08T11:56:02-05:00" level=info msg="Build context (go=go1.9.4, user=, date=)" source="mysqld_exporter.go:293"
time="2018-06-08T11:56:02-05:00" level=info msg="HTTPS/TLS is enabled" source="mysqld_exporter.go:362"
time="2018-06-08T11:56:02-05:00" level=info msg="Enabled High Resolution scrapers:" source="mysqld_exporter.go:376"
time="2018-06-08T11:56:02-05:00" level=info msg=" --collect.info_schema.innodb_metrics" source="mysqld_exporter.go:378"
time="2018-06-08T11:56:02-05:00" level=info msg=" --collect.global_status" source="mysqld_exporter.go:378"
time="2018-06-08T11:56:02-05:00" level=info msg="Enabled Medium Resolution scrapers:" source="mysqld_exporter.go:382"
time="2018-06-08T11:56:02-05:00" level=info msg=" --collect.info_schema.processlist" source="mysqld_exporter.go:384"
time="2018-06-08T11:56:02-05:00" level=info msg=" --collect.slave_status" source="mysqld_exporter.go:384"
time="2018-06-08T11:56:02-05:00" level=info msg=" --collect.perf_schema.eventswaits" source="mysqld_exporter.go:384"
time="2018-06-08T11:56:02-05:00" level=info msg=" --collect.perf_schema.file_events" source="mysqld_exporter.go:384"
time="2018-06-08T11:56:02-05:00" level=info msg=" --collect.info_schema.query_response_time" source="mysqld_exporter.go:384"
time="2018-06-08T11:56:02-05:00" level=info msg="Enabled Low Resolution scrapers:" source="mysqld_exporter.go:388"
time="2018-06-08T11:56:02-05:00" level=info msg=" --collect.global_variables" source="mysqld_exporter.go:390"
time="2018-06-08T11:56:02-05:00" level=info msg=" --collect.binlog_size" source="mysqld_exporter.go:390"
time="2018-06-08T11:56:02-05:00" level=info msg=" --collect.info_schema.userstats" source="mysqld_exporter.go:390"
time="2018-06-08T11:56:02-05:00" level=info msg="Listening on [server]:42002" source="mysqld_exporter.go:399"
time="2018-06-08T11:56:33-05:00" level=info msg="Starting mysqld_exporter (version=, branch=, revision=)" source="mysqld_exporter.go:292"
time="2018-06-08T11:56:33-05:00" level=info msg="Build context (go=go1.9.4, user=, date=)" source="mysqld_exporter.go:293"
time="2018-06-08T11:56:33-05:00" level=info msg="HTTPS/TLS is enabled" source="mysqld_exporter.go:362"
time="2018-06-08T11:56:33-05:00" level=info msg="Enabled High Resolution scrapers:" source="mysqld_exporter.go:376"
time="2018-06-08T11:56:33-05:00" level=info msg=" --collect.info_schema.innodb_metrics" source="mysqld_exporter.go:378"
time="2018-06-08T11:56:33-05:00" level=info msg=" --collect.global_status" source="mysqld_exporter.go:378"
time="2018-06-08T11:56:33-05:00" level=info msg="Enabled Medium Resolution scrapers:" source="mysqld_exporter.go:382"
time="2018-06-08T11:56:33-05:00" level=info msg=" --collect.info_schema.processlist" source="mysqld_exporter.go:384"
time="2018-06-08T11:56:33-05:00" level=info msg=" --collect.slave_status" source="mysqld_exporter.go:384"
time="2018-06-08T11:56:33-05:00" level=info msg=" --collect.perf_schema.eventswaits" source="mysqld_exporter.go:384"
time="2018-06-08T11:56:33-05:00" level=info msg=" --collect.perf_schema.file_events" source="mysqld_exporter.go:384"
time="2018-06-08T11:56:33-05:00" level=info msg=" --collect.info_schema.query_response_time" source="mysqld_exporter.go:384"
time="2018-06-08T11:56:33-05:00" level=info msg="Enabled Low Resolution scrapers:" source="mysqld_exporter.go:388"
time="2018-06-08T11:56:33-05:00" level=info msg=" --collect.global_variables" source="mysqld_exporter.go:390"
time="2018-06-08T11:56:33-05:00" level=info msg=" --collect.binlog_size" source="mysqld_exporter.go:390"
time="2018-06-08T11:56:33-05:00" level=info msg=" --collect.info_schema.userstats" source="mysqld_exporter.go:390"
time="2018-06-08T11:56:33-05:00" level=info msg="Listening on [server]:42002" source="mysqld_exporter.go:399"
2018/06/08 11:58:19 http: TLS handshake error from [server]:52196: tls: first record does not look like a TLS handshake
2018/06/08 12:20:33 http: Accept error: accept tcp [server]:42002: accept4: too many open files; retrying in 5ms
2018/06/08 12:20:33 http: Accept error: accept tcp [server]:42002: accept4: too many open files; retrying in 10ms
2018/06/08 12:20:33 http: Accept error: accept tcp [server]:42002: accept4: too many open files; retrying in 20ms
2018/06/08 12:20:33 http: Accept error: accept tcp [server]:42002: accept4: too many open files; retrying in 40ms
2018/06/08 12:20:33 http: Accept error: accept tcp [server]:42002: accept4: too many open files; retrying in 80ms
2018/06/08 12:20:33 http: Accept error: accept tcp [server]:42002: accept4: too many open files; retrying in 160ms
2018/06/08 12:20:33 http: Accept error: accept tcp [server]:42002: accept4: too many open files; retrying in 320ms

In prometheus dashboard for mysql metrics status is down.
Telnet is working fine and OS metrics is working, only mysql metrics it is giving this error.

Please help me in solving this.

Comments

Sign In or Register to comment.

MySQL, InnoDB, MariaDB and MongoDB are trademarks of their respective owners.
Copyright ©2005 - 2020 Percona LLC. All rights reserved.