Not the answer you need?
Register and ask your own question!

PMM OVA No Data Graphina

baaspbaasp ContributorCurrent User Role Novice
Hi,

We this week install the PMM OVA and tried connecting it to out lab MySql database servers.

However we see no data being published in graphina and no server / hosts to select.

Working with our telecomm team we have opened all ports in both directions and made a any to any relationship.

I do see data when I look at

http://192.168.145.100/prometheus/targets

https://192.168.142.138:42000/metrics

# HELP go_gc_duration_seconds A summary of the GC invocation durations. # TYPE go_gc_duration_seconds summary go_gc_duration_seconds{quantile="0"} 2.3994e-05 go_gc_duration_seconds{quantile="0.25"} 5.3523e-05 go_gc_duration_seconds{quantile="0.5"} 7.8095e-05 go_gc_duration_seconds{quantile="0.75"} 0.000821251 go_gc_duration_seconds{quantile="1"} 0.003140874 go_gc_duration_seconds_sum 399.803435443 go_gc_duration_seconds_count 577693 # HELP go_goroutines Number of goroutines that currently exist. # TYPE go_goroutines gauge go_goroutines 18 # HELP go_memstats_alloc_bytes Number of bytes allocated and still in use. # TYPE go_memstats_alloc_bytes gauge go_memstats_alloc_bytes 3.445456e+06 # HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed. # TYPE go_memstats_alloc_bytes_total counter go_memstats_alloc_bytes_total 1.720680342728e+12 # HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table. # TYPE go_memstats_buck_hash_sys_bytes gauge go_memstats_buck_hash_sys_bytes 1.857997e+06 # HELP go_memstats_frees_total Total number of frees. # TYPE go_memstats_frees_total counter go_memstats_frees_total 1.0925310175e+10 # HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata. # TYPE go_memstats_gc_sys_bytes gauge go_memstats_gc_sys_bytes 479232 # HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use. In the agent server pmm.log I see -bash-4.1$ cd /var/log -bash-4.1$ cat pmm-linux-metrics-42000.log time="2018-01-09T14:55:40-05:00" level=info msg="Starting node_exporter (version=0.14.0+percona.2, branch=master, revision=8ea8a4521f8f42d581847ee3d271dbb2a1fe8146)" source="node_exporter.go:142" time="2018-01-09T14:55:40-05:00" level=info msg="Build context (go=go1.9.2, [email protected], date=20171213-13:02:33)" source="node_exporter.go:143" time="2018-01-09T14:55:40-05:00" level=info msg="Enabled collectors:" source="node_exporter.go:162" time="2018-01-09T14:55:40-05:00" level=info msg=" - filefd" source="node_exporter.go:164" time="2018-01-09T14:55:40-05:00" level=info msg=" - loadavg" source="node_exporter.go:164" time="2018-01-09T14:55:40-05:00" level=info msg=" - netstat" source="node_exporter.go:164" time="2018-01-09T14:55:40-05:00" level=info msg=" - stat" source="node_exporter.go:164" time="2018-01-09T14:55:40-05:00" level=info msg=" - time" source="node_exporter.go:164" time="2018-01-09T14:55:40-05:00" level=info msg=" - vmstat" source="node_exporter.go:164" time="2018-01-09T14:55:40-05:00" level=info msg=" - diskstats" source="node_exporter.go:164" time="2018-01-09T14:55:40-05:00" level=info msg=" - filesystem" source="node_exporter.go:164" time="2018-01-09T14:55:40-05:00" level=info msg=" - meminfo" source="node_exporter.go:164" time="2018-01-09T14:55:40-05:00" level=info msg=" - netdev" source="node_exporter.go:164" time="2018-01-09T14:55:40-05:00" level=info msg=" - uname" source="node_exporter.go:164" time="2018-01-09T14:55:40-05:00" level=info msg="HTTP Basic authentication is enabled." source="basic_auth.go:105" time="2018-01-09T14:55:40-05:00" level=info msg="Starting HTTPS server of 192.168.142.138:42000 ..." source="server.go:106" 2018/01/09 14:55:52 http: TLS handshake error from 192.168.142.138:48312: tls: first record does not look like a TLS handshake 2018/01/09 14:55:52 http: TLS handshake error from 192.168.142.138:48314: tls: first record does not look like a TLS handshake 2018/01/09 14:59:04 http: TLS handshake error from 192.168.145.100:38744: tls: first record does not look like a TLS handshake 2018/01/09 15:35:05 http: TLS handshake error from 192.168.142.138:48472: tls: first record does not look like a TLS handshake 2018/01/09 15:35:05 http: TLS handshake error from 192.168.142.138:48474: tls: first record does not look like a TLS handshake 2018/01/11 08:23:28 http: TLS handshake error from 192.168.142.138:57224: tls: first record does not look like a TLS handshake 2018/01/11 08:23:28 http: TLS handshake error from 192.168.142.138:57222: tls: first record does not look like a TLS handshake 2018/01/11 11:32:23 http: TLS handshake error from 192.168.142.138:57894: tls: first record does not look like a TLS handshake 2018/01/11 11:32:23 http: TLS handshake error from 192.168.142.138:57896: tls: first record does not look like a TLS handshake on the agent server as well -bash-4.1$ sudo pmm-admin list pmm-admin 1.5.3 PMM Server | 192.168.145.100 (password-protected) Client Name | swcolabmysql01.wco.int Client Address | 192.168.142.138 Service Manager | unix-systemv





SERVICE TYPE NAME LOCAL PORT RUNNING DATA SOURCE OPTIONS





linux:metrics linux_swcolabmysql01 42000 YES - -bash-4.1$ sudo pmm-admin check-network –-no-emoji PMM Network Status Server Address | 192.168.145.100 Client Address | 192.168.142.138 * System Time PMM Server | 2018-01-07 20:44:32 +0000 GMT PMM Client | 2018-01-11 14:39:45 -0500 EST PMM Client to PMM Server Time Drift | 341713s Time is out of sync. Please make sure the server time is correct to see the metrics. * Connection: Client --> Server

SERVER SERVICE STATUS

Consul API OK Prometheus API OK Query Analytics API OK Connection duration | 1.866031ms Request duration | 1.498127ms Full round trip | 3.364158ms * Connection: Client <-- Server





SERVICE TYPE NAME REMOTE ENDPOINT STATUS HTTPS/TLS PASSWORD





linux:metrics linux_swcolabmysql01 192.168.142.138:42000 DOWN YES YES When an endpoint is down it may indicate that the corresponding service is stopped (run 'pmm-admin list' to verify). If it's running, check out the logs /var/log/pmm-*.log When all endpoints are down but 'pmm-admin list' shows they are up and no errors in the logs, check the firewall settings whether this system allows incoming connections from server to address:port in question. Also you can check the endpoint status by the URL: http://192.168.145.100/prometheus/targets

Comments

  • baaspbaasp Contributor Current User Role Novice
    On the PMM server in the pmm log we see [[email protected] log]# cat pmm* time="2018-01-07T17:38:16Z" level=info msg="Starting node_exporter (version=0.14.0+percona.2, branch=master, revision=8ea8a4521f8f42d581847ee3d271dbb2a1fe8146)" source="node_exporter.go:142" time="2018-01-07T17:38:16Z" level=info msg="Build context (go=go1.9.2, [email protected], date=20171130-13:11:09)" source="node_exporter.go:143" time="2018-01-07T17:38:16Z" level=info msg="Enabled collectors:" source="node_exporter.go:162" time="2018-01-07T17:38:16Z" level=info msg=" - diskstats" source="node_exporter.go:164" time="2018-01-07T17:38:16Z" level=info msg=" - loadavg" source="node_exporter.go:164" time="2018-01-07T17:38:16Z" level=info msg=" - netstat" source="node_exporter.go:164" time="2018-01-07T17:38:16Z" level=info msg=" - time" source="node_exporter.go:164" time="2018-01-07T17:38:16Z" level=info msg=" - stat" source="node_exporter.go:164" time="2018-01-07T17:38:16Z" level=info msg=" - uname" source="node_exporter.go:164" time="2018-01-07T17:38:16Z" level=info msg=" - vmstat" source="node_exporter.go:164" time="2018-01-07T17:38:16Z" level=info msg=" - filefd" source="node_exporter.go:164" time="2018-01-07T17:38:16Z" level=info msg=" - filesystem" source="node_exporter.go:164" time="2018-01-07T17:38:16Z" level=info msg=" - meminfo" source="node_exporter.go:164" time="2018-01-07T17:38:16Z" level=info msg=" - netdev" source="node_exporter.go:164" time="2018-01-07T17:38:16Z" level=info msg="HTTP Basic authentication is enabled." source="basic_auth.go:105" time="2018-01-07T17:38:16Z" level=info msg="Starting HTTPS server of 192.168.145.100:42000 ..." source="server.go:106" time="2017-12-05T13:29:14Z" level=info msg="PMM Configurator is started" address="127.0.0.1:7777" time="2017-12-05T13:30:15Z" level=info msg="PMM Server is configured correctly" time="2017-12-05T13:30:33Z" level=info msg="PMM Configurator is started" address="127.0.0.1:7777" time="2018-01-05T15:15:26Z" level=info msg="PMM Configurator is started" address="127.0.0.1:7777" time="2018-01-05T16:01:56Z" level=info msg="PMM Configurator is started" address="127.0.0.1:7777" time="2018-01-05T16:03:31Z" level=info msg="PMM Configurator is started" address="127.0.0.1:7777"
    as well on the pmm server

    [[email protected] log]# pmm-admin list
    pmm-admin 1.5.2

    PMM Server | 192.168.145.100 (password-protected)
    Client Name | scordbapmm01.wco.int
    Client Address | 192.168.145.100
    Service Manager | linux-systemd

    No services under monitoring.
    [[email protected] log]#
    [[email protected] log]#
    [[email protected] log]# pmm-admin check-network –-no-emoji
    PMM Network Status

    Server Address | 192.168.145.100
    Client Address | 192.168.145.100

    * System Time
    PMM Server | 2018-01-07 20:47:42 +0000 GMT
    PMM Client | 2018-01-07 20:47:47 +0000 UTC
    PMM Client to PMM Server Time Drift | OK

    * Connection: Client --> Server

    SERVER SERVICE STATUS

    Consul API OK
    Prometheus API OK
    Query Analytics API OK

    Connection duration | 451.754µs
    Request duration | 823.628µs
    Full round trip | 1.275382ms


    * Connection: Client <-- Server
    No metric endpoints registered.

    [[email protected] log]#



    As we are doing a POC to show the value of PMM and make the case to stop enterprise support with MySql Oracle I would really like to see this working.

    Please advise as to what next steps I should follow to correct this issue.

    Many thanks in advance.
  • PeterPeter Percona CEO Percona Moderator Role
    Hi,

    It does not looks like you're set up client correctly. What steps did you take to install pmm-agent ?

    You need something like "pmm-admin add mysql"

    https://www.percona.com/doc/percona-monitoring-and-management/pmm-admin.html#pmm-admin-add
  • baaspbaasp Contributor Current User Role Novice
    Hi,

    Yes I had initially done

    pmm-admin add mysql:metrics mysql_swcolabmysql01 --user pmm_admin --password PASSWORDHERE --host 192.168.###.###

    And there was no data.

    I then simplified and just tried for server metrics and still see no data in graphina.

    This is the issue, why do I nto see data from the monitored server?

    Thanks
  • baaspbaasp Contributor Current User Role Novice
    So to start back to beginning

    I have gone back to trying for the mysql data

    on the agent server I have

    -bash-4.1$ sudo pmm-admin add mysql:metrics mysql_swcolabmysql01 --user pmm_admin --password PASSWORD --host 192.168.142.138
    OK, now monitoring MySQL metrics using DSN pmm_admin:***@tcp(192.168.142.138:3306)
    -bash-4.1$
    -bash-4.1$
    -bash-4.1$
    -bash-4.1$ sudo pmm-admin list
    pmm-admin 1.5.3

    PMM Server | 192.168.145.100 (password-protected)
    Client Name | swcolabmysql01.wco.int
    Client Address | 192.168.142.138
    Service Manager | unix-systemv






    SERVICE TYPE NAME LOCAL PORT RUNNING DATA SOURCE OPTIONS





    linux:metrics linux_swcolabmysql01 42000 YES -
    mysql:metrics mysql_swcolabmysql01 42002 YES pmm_admin:***@tcp(192.168.142.138:3306)




    On the PMM Server I now have done


    [[email protected] ~]# pmm-admin list
    pmm-admin 1.5.2

    PMM Server | 192.168.145.100 (password-protected)
    Client Name | scordbapmm01.wco.int
    Client Address | 192.168.145.100
    Service Manager | linux-systemd

    No services under monitoring.
    [[email protected] ~]# pmm-admin add linux:metrics
    OK, now monitoring this system.
    [[email protected] ~]# pmm-admin list
    pmm-admin 1.5.2

    PMM Server | 192.168.145.100 (password-protected)
    Client Name | scordbapmm01.wco.int
    Client Address | 192.168.145.100
    Service Manager | linux-systemd






    SERVICE TYPE NAME LOCAL PORT RUNNING DATA SOURCE OPTIONS





    linux:metrics scordbapmm01.wco.int 42000 YES -
    [[email protected] ~]# pmm-admin list
    pmm-admin 1.5.2

    PMM Server | 192.168.145.100 (password-protected)
    Client Name | scordbapmm01.wco.int
    Client Address | 192.168.145.100
    Service Manager | linux-systemd






    SERVICE TYPE NAME LOCAL PORT RUNNING DATA SOURCE OPTIONS





    linux:metrics scordbapmm01.wco.int 42000 YES -
    [[email protected] ~]# pmm-admin list
    pmm-admin 1.5.2

    PMM Server | 192.168.145.100 (password-protected)
    Client Name | scordbapmm01.wco.int
    Client Address | 192.168.145.100
    Service Manager | linux-systemd






    SERVICE TYPE NAME LOCAL PORT RUNNING DATA SOURCE OPTIONS





    linux:metrics scordbapmm01.wco.int 42000 YES -
    [[email protected] ~]# pmm-admin add mysql
    [linux:metrics] OK, already monitoring this system.

    [mysql:metrics] OK, now monitoring MySQL metrics using DSN root:***@unix(/var/lib/mysql/mysql.sock)
    [mysql:queries] OK, now monitoring MySQL queries from slowlog using DSN root:***@unix(/var/lib/mysql/mysql.sock)
    [[email protected] ~]#
    [[email protected] ~]#
    [[email protected] ~]#
    [[email protected] ~]# pmm-admin list
    pmm-admin 1.5.2

    PMM Server | 192.168.145.100 (password-protected)
    Client Name | scordbapmm01.wco.int
    Client Address | 192.168.145.100
    Service Manager | linux-systemd






    SERVICE TYPE NAME LOCAL PORT RUNNING DATA SOURCE OPTIONS





    mysql:queries scordbapmm01.wco.int - YES root:***@unix(/var/lib/mysql/mysql.sock) query_source=slowlog, query_examples=true
    linux:metrics scordbapmm01.wco.int 42000 YES -
    mysql:metrics scordbapmm01.wco.int 42002 YES root:***@unix(/var/lib/mysql/mysql.sock)
  • baaspbaasp Contributor Current User Role Novice
    We see the agent being monitored but nothing in the graphina interface.
  • baaspbaasp Contributor Current User Role Novice
    And within 5 minutes I get this message on PMM Server

    [[email protected] ~]# pmm-admin list
    We have found system services disconnected from PMM server.
    Usually, this happens when data container is wiped before all monitoring services are removed or client is uninstalled.

    Orphaned local services: pmm-linux-metrics-42000, pmm-mysql-metrics-42002, pmm-mysql-queries-0

    To continue, run 'pmm-admin repair' to remove orphaned services.
  • baaspbaasp Contributor Current User Role Novice
    Ran repair

    [[email protected] ~]# pmm-admin repair
    OK, removed 3 orphaned services.
    [[email protected] ~]# pmm-admin list
    pmm-admin 1.5.2

    PMM Server | 192.168.145.100 (password-protected)
    Client Name | scordbapmm01.wco.int
    Client Address | 192.168.145.100
    Service Manager | linux-systemd

    No services under monitoring.


    Is clear now

    No errors iin PMM Server log

    [[email protected] ~]# cd /var/log
    [[email protected] log]# cat pmm-manage.log
    time="2017-12-05T13:29:14Z" level=info msg="PMM Configurator is started" address="127.0.0.1:7777"
    time="2017-12-05T13:30:15Z" level=info msg="PMM Server is configured correctly"
    time="2017-12-05T13:30:33Z" level=info msg="PMM Configurator is started" address="127.0.0.1:7777"
    time="2018-01-05T15:15:26Z" level=info msg="PMM Configurator is started" address="127.0.0.1:7777"
    time="2018-01-05T16:01:56Z" level=info msg="PMM Configurator is started" address="127.0.0.1:7777"
    time="2018-01-05T16:03:31Z" level=info msg="PMM Configurator is started" address="127.0.0.1:7777"
    [[email protected] log]# date
    Mon Jan 8 14:32:20 UTC 2018
  • baaspbaasp Contributor Current User Role Novice
    I will need assistance in further investigation. From what I see the lab server is being monitored. Just that the data is not being presented.
  • PeterPeter Percona CEO Percona Moderator Role
    Hi,

    I would check out 2 things

    1) Do you see data in the "Prometheus" dashboard and for pmm-server in "System Overview" - these should work even without setting up an agent, something like this
    https://pmmdemo.percona.com/graph/dashboard/db/system-overview?orgId=1&var-interval=$__auto_interval&var-host=pmm-server

    Do you see your other nodes here or not ?


    2) Go to the metrics URL https://metrics scordbapmm01.wco.int:42002/metrics-hr - do you see metrics here ? If your MySQL configuration is not correct you may have exporter running but not able to retrieve any metrics
  • baaspbaasp Contributor Current User Role Novice
    Wow. I rebooted the server and I now see data in graphina.

    Yay:)
  • PeterPeter Percona CEO Percona Moderator Role
    Great to hear!
  • baaspbaasp Contributor Current User Role Novice
    But while working when I run on PMM Server pmm-admin list I see this message

    [[email protected] ~]# pmm-admin list
    PMM server reports services that are missing locally.
    Usually, this happens when the system is completely reinstalled.

    Orphaned remote services: consul

    Beware, if another system with the same client name created those services, repairing the installation will remove remote services
    and the other system will be left with orphaned local services. If you are sure there is no other system with the same name,
    run 'pmm-admin repair' to remove orphaned services. Otherwise, please reinstall this client.
  • PeterPeter Percona CEO Percona Moderator Role
    Hi,

    Typically you would not install pmm-client on the PMM Server. It already has agent built in in the Server Docker Container.
  • baaspbaasp Contributor Current User Role Novice
    Ok, so I have done

    [[email protected] ~]# pmm-admin list
    PMM server reports services that are missing locally.
    Usually, this happens when the system is completely reinstalled.

    Orphaned remote services: consul

    Beware, if another system with the same client name created those services, repairing the installation will remove remote services
    and the other system will be left with orphaned local services. If you are sure there is no other system with the same name,
    run 'pmm-admin repair' to remove orphaned services. Otherwise, please reinstall this client.
    [[email protected] ~]# pmm-admin repair
    OK, removed 1 orphaned services.
    [[email protected] ~]# pmm-admin list
    pmm-admin 1.5.2

    PMM Server | 192.168.145.100 (password-protected)
    Client Name | scordbapmm01.wco.int
    Client Address | 192.168.145.100
    Service Manager | linux-systemd

    No services under monitoring.

    =========================

    Just to be clear, we do not need to run

    pmm-admin config

    On the PMM server?
  • PeterPeter Percona CEO Percona Moderator Role
    Right. You do not run pmm-admin on the PMM Server. I typically do not even install pmm-client on this server
Sign In or Register to comment.

MySQL, InnoDB, MariaDB and MongoDB are trademarks of their respective owners.
Copyright ©2005 - 2020 Percona LLC. All rights reserved.