I’ve done a couple of PMM docker installs and all seems to have worked fine. But for the first time we are running the docker PMM on the same host where the database is running as well (acceptance environment). And it doesn’t seem to be working correctly and I have no idea where to look for to solve the issue.
Setup:
Ubuntu 14.04 Host
Percona-mysql running on host
PMM-server is running in the docker on the host:
root@accdb01:~# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
7c7389a798dc percona/pmm-server:latest "/opt/entrypoint.sh" 21 hours ago Up 21 hours 443/tcp, 0.0.0.0:8585->80/tcp pmm-server
The pmm-admin shows it’s connected to the docker:
root@accdb01:~# pmm-admin list
pmm-admin 1.14.1
PMM Server | ip.add.re.ss:8585
Client Name | accdb01.xxxx
Client Address | 172.17.0.1
Service Manager | linux-upstart
-------------- ----------------------- ----------- -------- ------------------------------------------------------- --------------------------------------------------------------------------------------
SERVICE TYPE NAME LOCAL PORT RUNNING DATA SOURCE OPTIONS
-------------- ----------------------- ----------- -------- ------------------------------------------------------- --------------------------------------------------------------------------------------
mysql:queries accdb01.xxxx - YES debian-sys-maint:***@unix(/var/run/mysqld/mysqld.sock) query_source=slowlog, query_examples=true, slow_log_rotation=true, retain_slow_logs=1
linux:metrics accdb01.xxxx 42000 YES -
mysql:metrics accdb01.xxxx 42002 YES -
However if I open the PMM interface, it shows:
Monitored DB Instances
no value
If I open the query analytics, it does show queries the server has been serving, so that works, however if I open mysql graphs on the PMM client, it has no hosts and therefor doesn’t show any information.
Where should I start to look for what is causing the problem? Any checks I could perform to see what the problem might be?
Hi RBakkum
I have a similar configuration for our Staging builds here at Percona, except my build is based on Amazon Linux AMI (2017.09). Specifically we run a single EC2 instance with docker daemon for PMM Server, and then we run mysqld, mysqld_exporter, etc in non-docker way, in the EC2 instance.
Can you share with us the commands you used to add the host? For example:
pmm-admin add mysql
Also have you examined the log file for MySQL Metrics to see if there are any failures? The file is:
/var/log/pmm-mysql-metrics-42002.log
You should also try to fetch metrics from the exporter directly to validate whether the exporter is correctly configured, for example:
[root@ip-10-178-0-111 ~]# curl -sk https://10.178.0.111:42002/metrics-lr | head
# HELP mysql_binlog_file_number The last binlog file number.
# TYPE mysql_binlog_file_number gauge
mysql_binlog_file_number 1
[root@ip-10-178-0-111 ~]# pmm-admin list
pmm-admin 1.14.1
PMM Server | 10.178.0.111
Client Name | pmm-client-hostname
Client Address | 10.178.0.111
Service Manager | linux-upstart
-------------- ---------- ----------- -------- ----------------------------------- ---------------------------------------------
SERVICE TYPE NAME LOCAL PORT RUNNING DATA SOURCE OPTIONS
-------------- ---------- ----------- -------- ----------------------------------- ---------------------------------------------
mysql:queries PS_NODE-1 - YES root:***@unix(/tmp/PS_NODE_1.sock) query_source=perfschema, query_examples=true
linux:metrics PS_NODE-1 42000 YES -
mysql:metrics PS_NODE-1 42002 YES -
When I look at my targets, I can see they all use the EC2 host’s local IP address (10.x) - the entries with localhost are from inside the docker container:
The problem was in firewall not allowing docker traffic into the host. It was only allowed to forward traffic. So adjusting this to allow 42000/tcp and 42002/tcp solved my issue. The look into the targets helped me solve this, should have checked there as well last time
Just found it odd that the ‘pmm-admin list’ tells me that it’s alive, but in the end it wasn’t.
