PMM2 Setup - Everything is fine in the shell but prometheus says targets are down

Hello all,

Just started doing my upgrade to the version 2. Same machines that worked normally on verion 1. We’ve done a purge and install version2. My problem is that the single client I’ve tried to add so far does not show up.

Server is CentOS 7 and client is CentOS 6


pmm-admin status
Agent ID: /agent_id/07da9659-ffc9-400d-af5b-6577d5b928b8
Node ID : /node_id/b7fd1b5b-4d18-4eaf-a8b7-6bc6ef5bc63b

PMM Server:
URL : https://admin:admin@myserver.com:443/
Version: 2.0.0

PMM-agent:
Connected : true
Time drift: 2.960924ms
Latency : 29.398499ms

Agents:
/agent_id/67c8da01-2864-4276-a868-070d142d2708 NODE_EXPORTER RUNNING
/agent_id/67f14a81-bfce-42ec-83f2-84ca3c2decb6 MYSQLD_EXPORTER RUNNING
/agent_id/eae4d8bc-0343-49a1-82a8-5dcdb06f7ee5 QAN_MYSQL_SLOWLOG_AGENT RUNNING

pmm-admin list
Service type Service name Address and port Service ID
MySQL servername 127.0.0.1:3306 /service_id/8af5c499-0332-4523-8e29-632a7437a27d

Agent type Status Agent ID Service ID
pmm-agent connected /agent_id/07da9659-ffc9-400d-af5b-6577d5b928b8
node_exporter running /agent_id/67c8da01-2864-4276-a868-070d142d2708
mysqld_exporter running /agent_id/67f14a81-bfce-42ec-83f2-84ca3c2decb6 /service_id/8af5c499-0332-4523-8e29-632a7437a27d
qan-mysql-slowlog-agent running /agent_id/eae4d8bc-0343-49a1-82a8-5dcdb06f7ee5 /service_id/8af5c499-0332-4523-8e29-632a7437a27d

The thing that interests me is that the prometheus page is showing the client with it’s private ip as that target. Is it because there is a two way connection?

On the monitor server I ran the following command and it timed out.

docker exec -it pmm-server curl --insecure https://clientPublicOrPrivateIP:42000/metrics | head

I really need to figure this out first before I jump into my other clients and right now we are not monitoring anything since the server upgrade. Thank you for taking your time and looking. I appreciate it.

Why does prometheus go to http when I am doing everything over https? I’ve included my certs directory for the docker container, running it on 443, I don’t understand.

Also, it is trying to bind http://private.ip:42000, why? That’s not available to public thats probably why it can’t pull any data.