Not the answer you need?
Register and ask your own question!

Only Linux metrics are captured. MySQL metrics not available in the dashboard

Hi Guys,
I tried setting up PMM server via docker image and AWS AMI. I installed PMM client in my DB host.
I added mysql metrics with below command
pmm-admin add mysql --user username --password password

[[email protected] ~]# pmm-admin list
pmm-admin 1.17.0

PMM Server | 192.168.5.205 (password-protected)
Client Name | ip-192-168-5-203.us-east-2.compute.internal
Client Address | 192.168.5.203
Service Manager | linux-systemd






SERVICE TYPE NAME LOCAL PORT RUNNING DATA SOURCE OPTIONS





mysql:queries ip-192-168-5-203.us-east-2.compute.internal - YES pmm:***@unix(/var/lib/mysql/mysql.sock) query_source=slowlog, query_examples=true, slow_log_rotation=true, retain_slow_logs=1
linux:metrics ip-192-168-5-203.us-east-2.compute.internal 42000 YES -
mysql:metrics ip-192-168-5-203.us-east-2.compute.internal 42002 YES pmm:***@tcp(192.168.5.203:3306)

I can see the OS metrics in the dashboard but mysql metrics are not captured.
I opened ports 42000,42002 in my DB host already. PMM server is running in AWS T2 micro EC2 instance,since its a POC


[[email protected] ~]# pmm-admin check-network
PMM Network Status

Server Address | 192.168.5.205
Client Address | 192.168.5.203

* System Time
NTP Server (0.pool.ntp.org) | 2018-12-19 09:54:23 +0000 UTC
PMM Server | 2018-12-19 09:54:23 +0000 GMT
PMM Client | 2018-12-19 09:54:23 +0000 UTC
PMM Server Time Drift | OK
PMM Client Time Drift | OK
PMM Client to PMM Server Time Drift | OK

* Connection: Client --> Server

SERVER SERVICE STATUS

Consul API OK
Prometheus API OK
Query Analytics API OK

Connection duration | 458.015µs
Request duration | 133.959µs
Full round trip | 591.974µs


* Connection: Client <-- Server





SERVICE TYPE NAME REMOTE ENDPOINT STATUS HTTPS/TLS PASSWORD





linux:metrics ip-192-168-5-203.us-east-2.compute.internal 192.168.5.203:42000 OK YES YES
mysql:metrics ip-192-168-5-203.us-east-2.compute.internal 192.168.5.203:42002 DOWN YES YES

When an endpoint is down it may indicate that the corresponding service is stopped (run 'pmm-admin list' to verify).
If it's running, check out the logs /var/log/pmm-*.log

When all endpoints are down but 'pmm-admin list' shows they are up and no errors in the logs,
check the firewall settings whether this system allows incoming connections from server to address:port in question.

Also you can check the endpoint status by the URL: http://192.168.5.205/prometheus/targets

When i look at less /var/log/pmm-mysql-metrics-42002.log, i could see below error

2018/12/19 09:54:23 http: TLS handshake error from 192.168.5.203:39188: tls: first record does not look like a TLS handshake
2018/12/19 09:54:23 http: TLS handshake error from 192.168.5.203:39190: tls: first record does not look like a TLS handshake

Please help me in troubleshooting this issue.

Thanks

Comments

  • Michael CoburnMichael Coburn Principal Architect, Percona Percona Staff Role
    First, those TLS Handshake errors are safe to ignore.

    Now you want to confirm you can connect to the exporters correctly, so for example, from PMM Server run a command similar to this:
    curl -sk https://192.168.5.203:42002/metrics-hr

    Do you see records? You should on a working system.

    Let us know if this works or not, then we can continue troubleshooting
  • muffycompomuffycompo Entrant Current User Role Novice
    Hello Michael, we are having the same issue as mentioned above and running the curl command on the PMM server gave us an out "Invalid username or password". We can connect to the Web UI using the credentials we used when running the PMM server container and when we hit the URL in a web browser, we can authenticate using the correct username/password.
  • muffycompomuffycompo Entrant Current User Role Novice
    We have the same issue as mentioned above and after running the curl command on the PMM server, we got an out "Invalid username or password". We can hit the URL and can authenticate using the username/password we set when running PMM Server container.
  • Michael CoburnMichael Coburn Principal Architect, Percona Percona Staff Role
    muffycompo wrote: »
    Hello Michael, we are having the same issue as mentioned above and running the curl command on the PMM server gave us an out "Invalid username or password". We can connect to the Web UI using the credentials we used when running the PMM server container and when we hit the URL in a web browser, we can authenticate using the correct username/password.

    If you've configured username/password then you need curl to use it as well, here is an example:
    curl -u myuser:mypass -sk https://192.168.5.203:42002/metrics-hr
  • muffycompomuffycompo Entrant Current User Role Novice

    If you've configured username/password then you need curl to use it as well, here is an example:
    curl -u myuser:mypass -sk https://192.168.5.203:42002/metrics-hr

    Thanks for pointing that out Michael, It showed the content of the /metrics stats. Now it works but not sure what fixed it. All the endpoints showed DOWN but then after restarting the PMM Server server, they all came back as UP.
  • Michael CoburnMichael Coburn Principal Architect, Percona Percona Staff Role
    Interesting, not sure what changed, but I am glad you are recovered muffycompo !
  • udhayanudhayan Entrant Inactive User Role Novice
    Thanks for the reply Michael. I guess the problem was RAM of the docker server. Since it is T2 Ec2 instance , it had only 1 GB memory.
    I was able to set PMM without any issues in PROD setup with ample amount of resources.
  • Michael CoburnMichael Coburn Principal Architect, Percona Percona Staff Role
    Great udhayan , glad it is solved!
This discussion has been closed.

MySQL, InnoDB, MariaDB and MongoDB are trademarks of their respective owners.
Copyright ©2005 - 2020 Percona LLC. All rights reserved.