PMM doesn't show Mysql client added - nor does it appear on scrape

bombcar · February 14, 2021, 5:12pm

I’ve setup PMM on a server, and added a client, which correctly shows up under PMM inventory - but nothing appears in the MYSQL overview, and it doesn’t show on the prometheus scrape list.

Documentation refers to “pmm-admin check-network” but that just errors out with:

wiki mysql # pmm-admin check-network
pmm-admin: error: expected command but got "check-network", try --help

pmm-admin list shows:

wiki mysql # pmm-admin list
Service type                Service name                        Address and port       Service ID
MySQL                       wiki.example.com-mysql                  127.0.0.1:3306         /service_id/cbff4c44-2f09-49fb-a0cd-273c4093ee43

Agent type                  Status     Metrics Mode   Agent ID                                      Service ID
pmm_agent                   Connected    /agent_id/c49e2a69-33fc-4ddd-9e0f-ec02b7bc5ad8
node_exporter               Running    push  /agent_id/a6ecf473-cf6a-441c-853e-1bc630e96870
mysqld_exporter             Running    push  /agent_id/6b9eb3bd-5989-4dc2-b408-a579e1edb933 /service_id/cbff4c44-2f09-49fb-a0cd-273c4093ee43
mysql_slowlog_agent         Running      /agent_id/a3706303-1471-4150-8772-1aacbc83ccc3 /service_id/cbff4c44-2f09-49fb-a0cd-273c4093ee43
vmagent                     Running    push  /agent_id/8626cb6f-e2d4-4bd1-a1af-43dc6b3f3a72

Running curl from either the server or the client gets a large dump of data:

curl -k -u pmm:/agent_id/6b9eb3bd-5989-4dc2-b408-a579e1edb933 http://wiki.example.com:42002/metrics

adivinho · February 14, 2021, 5:11pm

Hi Tom,

“it doesn’t show on the prometheus scrape list.”

Please confirm that you have checked scrape targets through the next url
https://<your pmm server ip>/prometheus/targets

bombcar · February 14, 2021, 5:25pm

Yes - that’s the one - it shows various things (like a postgresql database) which I assume are the pmm server itself - but nothing about MySQL.

I installed it using the OVA image on VMWARE.

adivinho · February 15, 2021, 10:04am

No service instances are shown in VM targets if push mode is used.
Metrics are processed by local vmagent.

You may check amount of collected metrics for a particular service in Explore.

count({service_name="wiki.example.com-mysql"})

bombcar · February 15, 2021, 10:19am

I get: 0 series returned - if monitoring VM targets do I need to run PMM on the mysql server itself?

adivinho · February 15, 2021, 10:44am

Remotely located services can be monitored as well.

bombcar · February 15, 2021, 10:54am

I’m confused - the documentation is incorrect? You are supposed to add a remove server using that and NOT via pmm-admin on the remote system itself?

To be precise, I have PMM running on a host (pmm.example.com) and I ran the pmm-admin commands on wiki.example.com (another host) - both are VMs running Linux (not a “cloud database” if that matters). That remote server thing seems to be wanting to do remote MySQL commands across the internet? I guess I could do that but it seems … insecure.

adivinho · February 15, 2021, 11:10am

It has to be shown on PMM server side according to provided exporter statuses (Running).
Do you see any node metrics on dashboards for host wiki.example.com?

Could you perform pmm-admin summary command and check if any errors are occurred in logs?

bombcar · February 15, 2021, 11:17am

It appears the PMM server only sees itself. Attached are the summary ZIPs from both ends here: https://mef.box.com/s/qha3bl5qf57ebiwmw89owa3oyyo9usit

Looking in the errors indicates it may be an IPv6 issue - the client is trying to communicate with the IPv4 of the server which is a private IP? It should only be using the hostnames.

2021/02/14 13:51:17 [warn] 1139#1139: *10 an upstream response is buffered to a temporary file /var/cache/nginx/proxy_temp/1/00/0000000001 while reading upstream, client: 192.168.88.108, server: _, request: "GET /graph/public/img/login_background_dark.svg HTTP/1.1", upstream: "http://127.0.0.1:3000/public/img/login_background_dark.svg", host: "192.168.88.160", referrer: "http://192.168.88.160/graph/"

adivinho · February 15, 2021, 12:44pm

We may exclude such issues by specifying IP addresses in a configuration command.
Please notice that after a config command a mysql service has to be added for monitoring again.
e.g.
pmm-admin config --server-url="https://admin:K4xu5gDEKQhmsyxW@192.168.88.160" --server-insecure-tls 192.168.88.108 generic myNode

bombcar · February 15, 2021, 1:40pm

I have been using that but with the domain name of the IPv6 host - the IPv4 is unroutable. Should I be using the bracket form of the ipv6?