PMM doesn't show Mysql client added - nor does it appear on scrape

Percona Monitoring and Management (PMM) PMM 2.x
1

I’ve setup PMM on a server, and added a client, which correctly shows up under PMM inventory - but nothing appears in the MYSQL overview, and it doesn’t show on the prometheus scrape list.

Documentation refers to “pmm-admin check-network” but that just errors out with:

wiki mysql # pmm-admin check-network
pmm-admin: error: expected command but got "check-network", try --help

pmm-admin list shows:

wiki mysql # pmm-admin list
Service type                Service name                        Address and port       Service ID
MySQL                       wiki.example.com-mysql                  127.0.0.1:3306         /service_id/cbff4c44-2f09-49fb-a0cd-273c4093ee43

Agent type                  Status     Metrics Mode   Agent ID                                      Service ID
pmm_agent                   Connected    /agent_id/c49e2a69-33fc-4ddd-9e0f-ec02b7bc5ad8
node_exporter               Running    push  /agent_id/a6ecf473-cf6a-441c-853e-1bc630e96870
mysqld_exporter             Running    push  /agent_id/6b9eb3bd-5989-4dc2-b408-a579e1edb933 /service_id/cbff4c44-2f09-49fb-a0cd-273c4093ee43
mysql_slowlog_agent         Running      /agent_id/a3706303-1471-4150-8772-1aacbc83ccc3 /service_id/cbff4c44-2f09-49fb-a0cd-273c4093ee43
vmagent                     Running    push  /agent_id/8626cb6f-e2d4-4bd1-a1af-43dc6b3f3a72

Running curl from either the server or the client gets a large dump of data:

curl -k -u pmm:/agent_id/6b9eb3bd-5989-4dc2-b408-a579e1edb933 http://wiki.example.com:42002/metrics
2

Hi Tom,

“it doesn’t show on the prometheus scrape list.”

Please confirm that you have checked scrape targets through the next url
https://<your pmm server ip>/prometheus/targets

2 Likes
3

Yes - that’s the one - it shows various things (like a postgresql database) which I assume are the pmm server itself - but nothing about MySQL.

I installed it using the OVA image on VMWARE.

2 Likes
4

No service instances are shown in VM targets if push mode is used.
Metrics are processed by local vmagent.

You may check amount of collected metrics for a particular service in Explore.

count({service_name="wiki.example.com-mysql"})

Screenshot_20210215_115349
Screenshot_20210215_1153491846×354 24.3 KB

2 Likes
5

I get: 0 series returned - if monitoring VM targets do I need to run PMM on the mysql server itself?

2 Likes
6

Remotely located services can be monitored as well.

Screenshot_20210215_124013
Screenshot_20210215_1240131455×656 25.8 KB

2 Likes
7

I’m confused - the documentation is incorrect? You are supposed to add a remove server using that and NOT via pmm-admin on the remote system itself?

To be precise, I have PMM running on a host (pmm.example.com) and I ran the pmm-admin commands on wiki.example.com (another host) - both are VMs running Linux (not a “cloud database” if that matters). That remote server thing seems to be wanting to do remote MySQL commands across the internet? I guess I could do that but it seems … insecure.

2 Likes
8

It has to be shown on PMM server side according to provided exporter statuses (Running).
Do you see any node metrics on dashboards for host wiki.example.com?

Could you perform pmm-admin summary command and check if any errors are occurred in logs?

2 Likes
9

It appears the PMM server only sees itself. Attached are the summary ZIPs from both ends here: Box

Looking in the errors indicates it may be an IPv6 issue - the client is trying to communicate with the IPv4 of the server which is a private IP? It should only be using the hostnames.

2021/02/14 13:51:17 [warn] 1139#1139: *10 an upstream response is buffered to a temporary file /var/cache/nginx/proxy_temp/1/00/0000000001 while reading upstream, client: 192.168.88.108, server: _, request: "GET /graph/public/img/login_background_dark.svg HTTP/1.1", upstream: "http://127.0.0.1:3000/public/img/login_background_dark.svg", host: "192.168.88.160", referrer: "http://192.168.88.160/graph/"
2 Likes
10

We may exclude such issues by specifying IP addresses in a configuration command.
Please notice that after a config command a mysql service has to be added for monitoring again.
e.g.
pmm-admin config --server-url="https://admin:K4xu5gDEKQhmsyxW@192.168.88.160" --server-insecure-tls 192.168.88.108 generic myNode

2 Likes
11

I have been using that but with the domain name of the IPv6 host - the IPv4 is unroutable. Should I be using the bracket form of the ipv6?

2 Likes