Not the answer you need?
Register and ask your own question!

Pmm-client is alive, but not graphing

Hi,

I've done a couple of PMM docker installs and all seems to have worked fine. But for the first time we are running the docker PMM on the same host where the database is running as well (acceptance environment). And it doesn't seem to be working correctly and I have no idea where to look for to solve the issue.

Setup:
Ubuntu 14.04 Host
Percona-mysql running on host
PMM-server is running in the docker on the host:
```
[email protected]:~# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
7c7389a798dc percona/pmm-server:latest "/opt/entrypoint.sh" 21 hours ago Up 21 hours 443/tcp, 0.0.0.0:8585->80/tcp pmm-server
```

The pmm-admin shows it's connected to the docker:
```
[email protected]:~# pmm-admin list
pmm-admin 1.14.1

PMM Server | ip.add.re.ss:8585
Client Name | accdb01.xxxx
Client Address | 172.17.0.1
Service Manager | linux-upstart






SERVICE TYPE NAME LOCAL PORT RUNNING DATA SOURCE OPTIONS





mysql:queries accdb01.xxxx - YES debian-sys-maint:***@unix(/var/run/mysqld/mysqld.sock) query_source=slowlog, query_examples=true, slow_log_rotation=true, retain_slow_logs=1
linux:metrics accdb01.xxxx 42000 YES -
mysql:metrics accdb01.xxxx 42002 YES -
```

However if I open the PMM interface, it shows:
```
Monitored DB Instances
no value
```

If I open the query analytics, it does show queries the server has been serving, so that works, however if I open mysql graphs on the PMM client, it has no hosts and therefor doesn't show any information.

Where should I start to look for what is causing the problem? Any checks I could perform to see what the problem might be?

Comments

  • Michael CoburnMichael Coburn Principal Architect, Percona Percona Staff Role
    Hi RBakkum
    I have a similar configuration for our Staging builds here at Percona, except my build is based on Amazon Linux AMI (2017.09). Specifically we run a single EC2 instance with docker daemon for PMM Server, and then we run mysqld, mysqld_exporter, etc in non-docker way, in the EC2 instance.

    Can you share with us the commands you used to add the host? For example:
    pmm-admin add mysql
    
    Also have you examined the log file for MySQL Metrics to see if there are any failures? The file is:
    /var/log/pmm-mysql-metrics-42002.log
    
    You should also try to fetch metrics from the exporter directly to validate whether the exporter is correctly configured, for example:
    [root@ip-10-178-0-111 ~]# curl -sk https://10.178.0.111:42002/metrics-lr | head
    # HELP mysql_binlog_file_number The last binlog file number.
    # TYPE mysql_binlog_file_number gauge
    mysql_binlog_file_number 1
    
    [root@ip-10-178-0-111 ~]# pmm-admin list
    pmm-admin 1.14.1
    
    PMM Server      | 10.178.0.111
    Client Name     | pmm-client-hostname
    Client Address  | 10.178.0.111
    Service Manager | linux-upstart
    
    -------------- ---------- ----------- -------- ----------------------------------- ---------------------------------------------
    SERVICE TYPE   NAME       LOCAL PORT  RUNNING  DATA SOURCE                         OPTIONS
    -------------- ---------- ----------- -------- ----------------------------------- ---------------------------------------------
    mysql:queries  PS_NODE-1  -           YES      root:***@unix(/tmp/PS_NODE_1.sock)  query_source=perfschema, query_examples=true
    linux:metrics  PS_NODE-1  42000       YES      -
    mysql:metrics  PS_NODE-1  42002       YES      -
    

    When I look at my targets, I can see they all use the EC2 host's local IP address (10.x) - the entries with localhost are from inside the docker container:
  • RBakkumRBakkum Entrant Inactive User Role Novice
    Hi,

    Thanks for the help.

    The problem was in firewall not allowing docker traffic into the host. It was only allowed to forward traffic. So adjusting this to allow 42000/tcp and 42002/tcp solved my issue. The look into the targets helped me solve this, should have checked there as well last time :)
    Just found it odd that the 'pmm-admin list' tells me that it's alive, but in the end it wasn't.

    Just for reference (took me a while, for not a really grafana user):
    [URL="http://[pmm-host:port]/prometheus/targets"]http://[pmm-host: port]/prometheus/targets[/URL]
  • Michael CoburnMichael Coburn Principal Architect, Percona Percona Staff Role
    Hi RBakkum

    Glad to hear this is solved!
This discussion has been closed.

MySQL, InnoDB, MariaDB and MongoDB are trademarks of their respective owners.
Copyright ©2005 - 2020 Percona LLC. All rights reserved.