PMM OVA No Data Graphina

Percona Monitoring and Management (PMM) PMM 1.x
#1

Hi,

We this week install the PMM OVA and tried connecting it to out lab MySql database servers.

However we see no data being published in graphina and no server / hosts to select.

Working with our telecomm team we have opened all ports in both directions and made a any to any relationship.

I do see data when I look at

http://192.168.145.100/prometheus/targets

https://192.168.142.138:42000/metrics

HELP go_gc_duration_seconds A summary of the GC invocation durations. # TYPE go_gc_duration_seconds summary go_gc_duration_seconds{quantile=“0”} 2.3994e-05 go_gc_duration_seconds{quantile=“0.25”} 5.3523e-05 go_gc_duration_seconds{quantile=“0.5”} 7.8095e-05 go_gc_duration_seconds{quantile=“0.75”} 0.000821251 go_gc_duration_seconds{quantile=“1”} 0.003140874 go_gc_duration_seconds_sum 399.803435443 go_gc_duration_seconds_count 577693 # HELP go_goroutines Number of goroutines that currently exist. # TYPE go_goroutines gauge go_goroutines 18 # HELP go_memstats_alloc_bytes Number of bytes allocated and still in use. # TYPE go_memstats_alloc_bytes gauge go_memstats_alloc_bytes 3.445456e+06 # HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed. # TYPE go_memstats_alloc_bytes_total counter go_memstats_alloc_bytes_total 1.720680342728e+12 # HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table. # TYPE go_memstats_buck_hash_sys_bytes gauge go_memstats_buck_hash_sys_bytes 1.857997e+06 # HELP go_memstats_frees_total Total number of frees. # TYPE go_memstats_frees_total counter go_memstats_frees_total 1.0925310175e+10 # HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata. # TYPE go_memstats_gc_sys_bytes gauge go_memstats_gc_sys_bytes 479232 # HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use. In the agent server pmm.log I see -bash-4.1$ cd /var/log -bash-4.1$ cat pmm-linux-metrics-42000.log time=“2018-01-09T14:55:40-05:00” level=info msg=“Starting node_exporter (version=0.14.0+percona.2, branch=master, revision=8ea8a4521f8f42d581847ee3d271dbb2a1fe8146)” source=“node_exporter.go:142” time=“2018-01-09T14:55:40-05:00” level=info msg=“Build context (go=go1.9.2, user=jenkins@os-centos-6-x64-5062, date=20171213-13:02:33)” source=“node_exporter.go:143” time=“2018-01-09T14:55:40-05:00” level=info msg=“Enabled collectors:” source=“node_exporter.go:162” time=“2018-01-09T14:55:40-05:00” level=info msg=" - filefd" source=“node_exporter.go:164” time=“2018-01-09T14:55:40-05:00” level=info msg=" - loadavg" source=“node_exporter.go:164” time=“2018-01-09T14:55:40-05:00” level=info msg=" - netstat" source=“node_exporter.go:164” time=“2018-01-09T14:55:40-05:00” level=info msg=" - stat" source=“node_exporter.go:164” time=“2018-01-09T14:55:40-05:00” level=info msg=" - time" source=“node_exporter.go:164” time=“2018-01-09T14:55:40-05:00” level=info msg=" - vmstat" source=“node_exporter.go:164” time=“2018-01-09T14:55:40-05:00” level=info msg=" - diskstats" source=“node_exporter.go:164” time=“2018-01-09T14:55:40-05:00” level=info msg=" - filesystem" source=“node_exporter.go:164” time=“2018-01-09T14:55:40-05:00” level=info msg=" - meminfo" source=“node_exporter.go:164” time=“2018-01-09T14:55:40-05:00” level=info msg=" - netdev" source=“node_exporter.go:164” time=“2018-01-09T14:55:40-05:00” level=info msg=" - uname" source=“node_exporter.go:164” time=“2018-01-09T14:55:40-05:00” level=info msg=“HTTP Basic authentication is enabled.” source=“basic_auth.go:105” time=“2018-01-09T14:55:40-05:00” level=info msg=“Starting HTTPS server of 192.168.142.138:42000 …” source=“server.go:106” 2018/01/09 14:55:52 http: TLS handshake error from 192.168.142.138:48312: tls: first record does not look like a TLS handshake 2018/01/09 14:55:52 http: TLS handshake error from 192.168.142.138:48314: tls: first record does not look like a TLS handshake 2018/01/09 14:59:04 http: TLS handshake error from 192.168.145.100:38744: tls: first record does not look like a TLS handshake 2018/01/09 15:35:05 http: TLS handshake error from 192.168.142.138:48472: tls: first record does not look like a TLS handshake 2018/01/09 15:35:05 http: TLS handshake error from 192.168.142.138:48474: tls: first record does not look like a TLS handshake 2018/01/11 08:23:28 http: TLS handshake error from 192.168.142.138:57224: tls: first record does not look like a TLS handshake 2018/01/11 08:23:28 http: TLS handshake error from 192.168.142.138:57222: tls: first record does not look like a TLS handshake 2018/01/11 11:32:23 http: TLS handshake error from 192.168.142.138:57894: tls: first record does not look like a TLS handshake 2018/01/11 11:32:23 http: TLS handshake error from 192.168.142.138:57896: tls: first record does not look like a TLS handshake on the agent server as well -bash-4.1$ sudo pmm-admin list pmm-admin 1.5.3 PMM Server | 192.168.145.100 (password-protected) Client Name | swcolabmysql01.wco.int Client Address | 192.168.142.138 Service Manager | unix-systemv -------------- --------------------- ----------- -------- ------------ -------- SERVICE TYPE NAME LOCAL PORT RUNNING DATA SOURCE OPTIONS -------------- --------------------- ----------- -------- ------------ -------- linux:metrics linux_swcolabmysql01 42000 YES - -bash-4.1$ sudo pmm-admin check-network –-no-emoji PMM Network Status Server Address | 192.168.145.100 Client Address | 192.168.142.138 * System Time PMM Server | 2018-01-07 20:44:32 +0000 GMT PMM Client | 2018-01-11 14:39:45 -0500 EST PMM Client to PMM Server Time Drift | 341713s Time is out of sync. Please make sure the server time is correct to see the metrics. * Connection: Client --> Server -------------------- ------- SERVER SERVICE STATUS -------------------- ------- Consul API OK Prometheus API OK Query Analytics API OK Connection duration | 1.866031ms Request duration | 1.498127ms Full round trip | 3.364158ms * Connection: Client <-- Server -------------- --------------------- ---------------------- ------- ---------- --------- SERVICE TYPE NAME REMOTE ENDPOINT STATUS HTTPS/TLS PASSWORD -------------- --------------------- ---------------------- ------- ---------- --------- linux:metrics linux_swcolabmysql01 192.168.142.138:42000 DOWN YES YES When an endpoint is down it may indicate that the corresponding service is stopped (run ‘pmm-admin list’ to verify). If it’s running, check out the logs /var/log/pmm-*.log When all endpoints are down but ‘pmm-admin list’ shows they are up and no errors in the logs, check the firewall settings whether this system allows incoming connections from server to address:port in question. Also you can check the endpoint status by the URL: http://192.168.145.100/prometheus/targets

#2

On the PMM server in the pmm log we see [root@scordbapmm01 log]# cat pmm* time=“2018-01-07T17:38:16Z” level=info msg=“Starting node_exporter (version=0.14.0+percona.2, branch=master, revision=8ea8a4521f8f42d581847ee3d271dbb2a1fe8146)” source=“node_exporter.go:142” time=“2018-01-07T17:38:16Z” level=info msg=“Build context (go=go1.9.2, user=jenkins@os-centos-6-x64-2404, date=20171130-13:11:09)” source=“node_exporter.go:143” time=“2018-01-07T17:38:16Z” level=info msg=“Enabled collectors:” source=“node_exporter.go:162” time=“2018-01-07T17:38:16Z” level=info msg=" - diskstats" source=“node_exporter.go:164” time=“2018-01-07T17:38:16Z” level=info msg=" - loadavg" source=“node_exporter.go:164” time=“2018-01-07T17:38:16Z” level=info msg=" - netstat" source=“node_exporter.go:164” time=“2018-01-07T17:38:16Z” level=info msg=" - time" source=“node_exporter.go:164” time=“2018-01-07T17:38:16Z” level=info msg=" - stat" source=“node_exporter.go:164” time=“2018-01-07T17:38:16Z” level=info msg=" - uname" source=“node_exporter.go:164” time=“2018-01-07T17:38:16Z” level=info msg=" - vmstat" source=“node_exporter.go:164” time=“2018-01-07T17:38:16Z” level=info msg=" - filefd" source=“node_exporter.go:164” time=“2018-01-07T17:38:16Z” level=info msg=" - filesystem" source=“node_exporter.go:164” time=“2018-01-07T17:38:16Z” level=info msg=" - meminfo" source=“node_exporter.go:164” time=“2018-01-07T17:38:16Z” level=info msg=" - netdev" source=“node_exporter.go:164” time=“2018-01-07T17:38:16Z” level=info msg=“HTTP Basic authentication is enabled.” source=“basic_auth.go:105” time=“2018-01-07T17:38:16Z” level=info msg=“Starting HTTPS server of 192.168.145.100:42000 …” source=“server.go:106” time=“2017-12-05T13:29:14Z” level=info msg=“PMM Configurator is started” address=“127.0.0.1:7777” time=“2017-12-05T13:30:15Z” level=info msg=“PMM Server is configured correctly” time=“2017-12-05T13:30:33Z” level=info msg=“PMM Configurator is started” address=“127.0.0.1:7777” time=“2018-01-05T15:15:26Z” level=info msg=“PMM Configurator is started” address=“127.0.0.1:7777” time=“2018-01-05T16:01:56Z” level=info msg=“PMM Configurator is started” address=“127.0.0.1:7777” time=“2018-01-05T16:03:31Z” level=info msg=“PMM Configurator is started” address=“127.0.0.1:7777”
as well on the pmm server

[root@scordbapmm01 log]# pmm-admin list
pmm-admin 1.5.2

PMM Server | 192.168.145.100 (password-protected)
Client Name | scordbapmm01.wco.int
Client Address | 192.168.145.100
Service Manager | linux-systemd

No services under monitoring.
[root@scordbapmm01 log]#
[root@scordbapmm01 log]#
[root@scordbapmm01 log]# pmm-admin check-network –-no-emoji
PMM Network Status

Server Address | 192.168.145.100
Client Address | 192.168.145.100

  • System Time
    PMM Server | 2018-01-07 20:47:42 +0000 GMT
    PMM Client | 2018-01-07 20:47:47 +0000 UTC
    PMM Client to PMM Server Time Drift | OK

  • Connection: Client --> Server

SERVER SERVICE STATUS

Consul API OK
Prometheus API OK
Query Analytics API OK

Connection duration | 451.754µs
Request duration | 823.628µs
Full round trip | 1.275382ms

  • Connection: Client <-- Server
    No metric endpoints registered.

[root@scordbapmm01 log]#

As we are doing a POC to show the value of PMM and make the case to stop enterprise support with MySql Oracle I would really like to see this working.

Please advise as to what next steps I should follow to correct this issue.

Many thanks in advance.

#3

Hi,

It does not looks like you’re set up client correctly. What steps did you take to install pmm-agent ?

You need something like “pmm-admin add mysql”

https://www.percona.com/doc/percona-monitoring-and-management/pmm-admin.html#pmm-admin-add

#4

Hi,

Yes I had initially done

pmm-admin add mysql:metrics mysql_swcolabmysql01 --user pmm_admin --password PASSWORDHERE --host 192.168.###.###

And there was no data.

I then simplified and just tried for server metrics and still see no data in graphina.

This is the issue, why do I nto see data from the monitored server?

Thanks

#5

So to start back to beginning

I have gone back to trying for the mysql data

on the agent server I have

-bash-4.1$ sudo pmm-admin add mysql:metrics mysql_swcolabmysql01 --user pmm_admin --password PASSWORD --host 192.168.142.138
OK, now monitoring MySQL metrics using DSN pmm_admin:***@tcp(192.168.142.138:3306)
-bash-4.1$
-bash-4.1$
-bash-4.1$
-bash-4.1$ sudo pmm-admin list
pmm-admin 1.5.3

PMM Server | 192.168.145.100 (password-protected)
Client Name | swcolabmysql01.wco.int
Client Address | 192.168.142.138
Service Manager | unix-systemv

SERVICE TYPE NAME LOCAL PORT RUNNING DATA SOURCE OPTIONS

linux:metrics linux_swcolabmysql01 42000 YES -
mysql:metrics mysql_swcolabmysql01 42002 YES pmm_admin:***@tcp(192.168.142.138:3306)

On the PMM Server I now have done

[root@scordbapmm01 ~]# pmm-admin list
pmm-admin 1.5.2

PMM Server | 192.168.145.100 (password-protected)
Client Name | scordbapmm01.wco.int
Client Address | 192.168.145.100
Service Manager | linux-systemd

No services under monitoring.
[root@scordbapmm01 ~]# pmm-admin add linux:metrics
OK, now monitoring this system.
[root@scordbapmm01 ~]# pmm-admin list
pmm-admin 1.5.2

PMM Server | 192.168.145.100 (password-protected)
Client Name | scordbapmm01.wco.int
Client Address | 192.168.145.100
Service Manager | linux-systemd

SERVICE TYPE NAME LOCAL PORT RUNNING DATA SOURCE OPTIONS

linux:metrics scordbapmm01.wco.int 42000 YES -
[root@scordbapmm01 ~]# pmm-admin list
pmm-admin 1.5.2

PMM Server | 192.168.145.100 (password-protected)
Client Name | scordbapmm01.wco.int
Client Address | 192.168.145.100
Service Manager | linux-systemd

SERVICE TYPE NAME LOCAL PORT RUNNING DATA SOURCE OPTIONS

linux:metrics scordbapmm01.wco.int 42000 YES -
[root@scordbapmm01 ~]# pmm-admin list
pmm-admin 1.5.2

PMM Server | 192.168.145.100 (password-protected)
Client Name | scordbapmm01.wco.int
Client Address | 192.168.145.100
Service Manager | linux-systemd

SERVICE TYPE NAME LOCAL PORT RUNNING DATA SOURCE OPTIONS

linux:metrics scordbapmm01.wco.int 42000 YES -
[root@scordbapmm01 ~]# pmm-admin add mysql
[linux:metrics] OK, already monitoring this system.

[mysql:metrics] OK, now monitoring MySQL metrics using DSN root:@unix(/var/lib/mysql/mysql.sock)
[mysql:queries] OK, now monitoring MySQL queries from slowlog using DSN root:@unix(/var/lib/mysql/mysql.sock)
[root@scordbapmm01 ~]#
[root@scordbapmm01 ~]#
[root@scordbapmm01 ~]#
[root@scordbapmm01 ~]# pmm-admin list
pmm-admin 1.5.2

PMM Server | 192.168.145.100 (password-protected)
Client Name | scordbapmm01.wco.int
Client Address | 192.168.145.100
Service Manager | linux-systemd

SERVICE TYPE NAME LOCAL PORT RUNNING DATA SOURCE OPTIONS

mysql:queries scordbapmm01.wco.int - YES root:@unix(/var/lib/mysql/mysql.sock) query_source=slowlog, query_examples=true
linux:metrics scordbapmm01.wco.int 42000 YES -
mysql:metrics scordbapmm01.wco.int 42002 YES root:@unix(/var/lib/mysql/mysql.sock)

#6

We see the agent being monitored but nothing in the graphina interface.

1772×720 124 KB

1666×901 134 KB

1696×924 131 KB

#7

And within 5 minutes I get this message on PMM Server

[root@scordbapmm01 ~]# pmm-admin list
We have found system services disconnected from PMM server.
Usually, this happens when data container is wiped before all monitoring services are removed or client is uninstalled.

Orphaned local services: pmm-linux-metrics-42000, pmm-mysql-metrics-42002, pmm-mysql-queries-0

To continue, run ‘pmm-admin repair’ to remove orphaned services.

#8

Ran repair

[root@scordbapmm01 ~]# pmm-admin repair
OK, removed 3 orphaned services.
[root@scordbapmm01 ~]# pmm-admin list
pmm-admin 1.5.2

PMM Server | 192.168.145.100 (password-protected)
Client Name | scordbapmm01.wco.int
Client Address | 192.168.145.100
Service Manager | linux-systemd

No services under monitoring.

Is clear now

No errors iin PMM Server log

[root@scordbapmm01 ~]# cd /var/log
[root@scordbapmm01 log]# cat pmm-manage.log
time=“2017-12-05T13:29:14Z” level=info msg=“PMM Configurator is started” address=“127.0.0.1:7777”
time=“2017-12-05T13:30:15Z” level=info msg=“PMM Server is configured correctly”
time=“2017-12-05T13:30:33Z” level=info msg=“PMM Configurator is started” address=“127.0.0.1:7777”
time=“2018-01-05T15:15:26Z” level=info msg=“PMM Configurator is started” address=“127.0.0.1:7777”
time=“2018-01-05T16:01:56Z” level=info msg=“PMM Configurator is started” address=“127.0.0.1:7777”
time=“2018-01-05T16:03:31Z” level=info msg=“PMM Configurator is started” address=“127.0.0.1:7777”
[root@scordbapmm01 log]# date
Mon Jan 8 14:32:20 UTC 2018

#9

I will need assistance in further investigation. From what I see the lab server is being monitored. Just that the data is not being presented.

1866×645 122 KB

#10

Hi,

I would check out 2 things

  1. Do you see data in the “Prometheus” dashboard and for pmm-server in “System Overview” - these should work even without setting up an agent, something like this
    https://pmmdemo.percona.com/graph/dashboard/db/system-overview?orgId=1&var-interval=$__auto_interval&var-host=pmm-server

Do you see your other nodes here or not ?

  1. Go to the metrics URL https://metrics scordbapmm01.wco.int:42002/metrics-hr - do you see metrics here ? If your MySQL configuration is not correct you may have exporter running but not able to retrieve any metrics
#11

Wow. I rebooted the server and I now see data in graphina.

Yay:)

#12

Great to hear!

#13

But while working when I run on PMM Server pmm-admin list I see this message

[root@scordbapmm01 ~]# pmm-admin list
PMM server reports services that are missing locally.
Usually, this happens when the system is completely reinstalled.

Orphaned remote services: consul

Beware, if another system with the same client name created those services, repairing the installation will remove remote services
and the other system will be left with orphaned local services. If you are sure there is no other system with the same name,
run ‘pmm-admin repair’ to remove orphaned services. Otherwise, please reinstall this client.

#14

Hi,

Typically you would not install pmm-client on the PMM Server. It already has agent built in in the Server Docker Container.

#15

Ok, so I have done

[root@scordbapmm01 ~]# pmm-admin list
PMM server reports services that are missing locally.
Usually, this happens when the system is completely reinstalled.

Orphaned remote services: consul

Beware, if another system with the same client name created those services, repairing the installation will remove remote services
and the other system will be left with orphaned local services. If you are sure there is no other system with the same name,
run ‘pmm-admin repair’ to remove orphaned services. Otherwise, please reinstall this client.
[root@scordbapmm01 ~]# pmm-admin repair
OK, removed 1 orphaned services.
[root@scordbapmm01 ~]# pmm-admin list
pmm-admin 1.5.2

PMM Server | 192.168.145.100 (password-protected)
Client Name | scordbapmm01.wco.int
Client Address | 192.168.145.100
Service Manager | linux-systemd

No services under monitoring.

=========================

Just to be clear, we do not need to run

pmm-admin config

On the PMM server?

#16

Right. You do not run pmm-admin on the PMM Server. I typically do not even install pmm-client on this server

MySQL, InnoDB, MariaDB and MongoDB are trademarks of their respective owners.
Copyright © 2006 - 2021 Percona LLC. All rights reserved.