PMM request error/template init failed etc.

We setup PMM server as a VM and installed pmm-client on one of our MariaDB servers. Configured pmm-server and the client node but the PMM graphs does not populate anything. It neither shows the host that has pmm-client installed.
When opening Metrics Monitor page, it gives “Templating init failed” and “Request Error” for each graph panel. When opening QAN page, it gives “QAN API error”. Attached screenshots of these errors.

The pmm client log has these messages:

2017/09/06 10:07:43.635269 WARNING data-sender Cannot connect to API: bad status
2017/09/06 10:08:20.076584 WARNING log-ws bad status
2017/09/06 10:08:20.076690 WARNING agent-ws bad status
2017/09/06 10:08:50.631109 WARNING data-sender Cannot connect to API: websocket.Dial ws://10.187.79.196:80/qan-api/agents/862188b1790d449a5b12ea563d6d4208/data: dial tcp 10.187.79.196:80: i/o timeout
2017/09/06 10:09:00.631718 WARNING data-sender Cannot connect to API: websocket.Dial ws://10.187.79.196:80/qan-api/agents/862188b1790d449a5b12ea563d6d4208/data: dial tcp 10.187.79.196:80: getsockopt: connection refused
2017/09/06 10:09:03.632574 WARNING data-sender Cannot connect to API: websocket.Dial ws://10.187.79.196:80/qan-api/agents/862188b1790d449a5b12ea563d6d4208/data: dial tcp 10.187.79.196:80: getsockopt: connection refused

When I try to connect to PMM server QAN API, I get this error:

[user@pmm-client]$ curl http://user:password@10.187.79.196:80/qan-api/
<html>
<head><title>502 Bad Gateway</title></head>
<body bgcolor="white">
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx</center>
</body>
</html>

Any pointers to solve this? Thanks!

photoid=49556

Can you run these and then attach the files here

To be executed on PMM server:
wget [URL=“https://raw.githubusercontent.com/Percona-QA/percona-qa/master/pmm-info.sh”]https://raw.githubusercontent.com/Pe...er/pmm-info.sh[/URL]
sh ./pmm-info.sh ext > pmm-server.log
To be executed on client server:
wget [URL=“https://raw.githubusercontent.com/Percona-QA/percona-qa/master/pmm-info.sh”]https://raw.githubusercontent.com/Pe...er/pmm-info.sh[/URL]
sh ./pmm-info.sh > pmm-client.log

Attached the output of the above.
Note: Our PMM server was imported as a VM from OVA and not a docker.

pmm-server.txt (2.58 KB)

pmm-client.txt (4.67 KB)

This is resolved. Couple of problems:

  1. Prometheus service was not running in PMM Server. Started it (nohup /usr/sbin/prometheus -config.file=/etc/prometheus.yml > prometheus.log &) and also set it up as a service
  2. Datasource URL for Prometheus was pointing to wrong link (by default). Modified to point to the correct port (http://localhost:9090)

Now the problem is with Alerting. It is not triggering an alert even after exceeding the threshold. I will probably open a separate discussion.