Only Linux metrics are captured. MySQL metrics not available in the dashboard

Percona Monitoring and Management (PMM) PMM 1.x
1

Hi Guys,
I tried setting up PMM server via docker image and AWS AMI. I installed PMM client in my DB host.
I added mysql metrics with below command
pmm-admin add mysql --user username --password password

[root@ip-192-168-5-203 ~]# pmm-admin list
pmm-admin 1.17.0

PMM Server | 192.168.5.205 (password-protected)
Client Name | ip-192-168-5-203.us-east-2.compute.internal
Client Address | 192.168.5.203
Service Manager | linux-systemd

SERVICE TYPE NAME LOCAL PORT RUNNING DATA SOURCE OPTIONS

mysql:queries ip-192-168-5-203.us-east-2.compute.internal - YES pmm:@unix(/var/lib/mysql/mysql.sock) query_source=slowlog, query_examples=true, slow_log_rotation=true, retain_slow_logs=1
linux:metrics ip-192-168-5-203.us-east-2.compute.internal 42000 YES -
mysql:metrics ip-192-168-5-203.us-east-2.compute.internal 42002 YES pmm:@tcp(192.168.5.203:3306)

I can see the OS metrics in the dashboard but mysql metrics are not captured.
I opened ports 42000,42002 in my DB host already. PMM server is running in AWS T2 micro EC2 instance,since its a POC

[root@ip-192-168-5-203 ~]# pmm-admin check-network
PMM Network Status

Server Address | 192.168.5.205
Client Address | 192.168.5.203

  • System Time
    NTP Server (0.pool.ntp.org) | 2018-12-19 09:54:23 +0000 UTC
    PMM Server | 2018-12-19 09:54:23 +0000 GMT
    PMM Client | 2018-12-19 09:54:23 +0000 UTC
    PMM Server Time Drift | OK
    PMM Client Time Drift | OK
    PMM Client to PMM Server Time Drift | OK

  • Connection: Client → Server

SERVER SERVICE STATUS

Consul API OK
Prometheus API OK
Query Analytics API OK

Connection duration | 458.015µs
Request duration | 133.959µs
Full round trip | 591.974µs

  • Connection: Client ← Server

SERVICE TYPE NAME REMOTE ENDPOINT STATUS HTTPS/TLS PASSWORD

linux:metrics ip-192-168-5-203.us-east-2.compute.internal 192.168.5.203:42000 OK YES YES
mysql:metrics ip-192-168-5-203.us-east-2.compute.internal 192.168.5.203:42002 DOWN YES YES

When an endpoint is down it may indicate that the corresponding service is stopped (run ‘pmm-admin list’ to verify).
If it’s running, check out the logs /var/log/pmm-*.log

When all endpoints are down but ‘pmm-admin list’ shows they are up and no errors in the logs,
check the firewall settings whether this system allows incoming connections from server to address:port in question.

Also you can check the endpoint status by the URL: [url]http://192.168.5.205/prometheus/targets[/url]

When i look at less /var/log/pmm-mysql-metrics-42002.log, i could see below error

2018/12/19 09:54:23 http: TLS handshake error from 192.168.5.203:39188: tls: first record does not look like a TLS handshake
2018/12/19 09:54:23 http: TLS handshake error from 192.168.5.203:39190: tls: first record does not look like a TLS handshake

Please help me in troubleshooting this issue.

Thanks

2

First, those TLS Handshake errors are safe to ignore.

Now you want to confirm you can connect to the exporters correctly, so for example, from PMM Server run a command similar to this:
curl -sk [url]https://192.168.5.203:42002/metrics-hr[/url]

Do you see records? You should on a working system.

Let us know if this works or not, then we can continue troubleshooting

3

Hello Michael, we are having the same issue as mentioned above and running the curl command on the PMM server gave us an out “Invalid username or password”. We can connect to the Web UI using the credentials we used when running the PMM server container and when we hit the URL in a web browser, we can authenticate using the correct username/password.

4

We have the same issue as mentioned above and after running the curl command on the PMM server, we got an out “Invalid username or password”. We can hit the URL and can authenticate using the username/password we set when running PMM Server container.

5

If you’ve configured username/password then you need curl to use it as well, here is an example:
curl -u myuser:mypass -sk https://192.168.5.203:42002/metrics-hr

6

Thanks for pointing that out Michael, It showed the content of the /metrics stats. Now it works but not sure what fixed it. All the endpoints showed DOWN but then after restarting the PMM Server server, they all came back as UP.

7

Interesting, not sure what changed, but I am glad you are recovered muffycompo !

8

Thanks for the reply Michael. I guess the problem was RAM of the docker server. Since it is T2 Ec2 instance , it had only 1 GB memory.
I was able to set PMM without any issues in PROD setup with ample amount of resources.

9

Great udhayan , glad it is solved!