Error adding MySQL queries: timeout 10s waiting on agent to connect to API

Percona Monitoring and Management (PMM) PMM 1.x
1

Hello All,

I’m trying to set up pmm monitoring, mostly for QAN but that part doesn’t seem to work. What could I be missing here?

pmm-admin add mysql --user pmm --password pass

[linux:metrics] OK, already monitoring this system.
[mysql:metrics] OK, already monitoring MySQL metrics.
[mysql:queries] Error adding MySQL queries: timeout 10s waiting on agent to connect to API

the pmm-queries log is also filled with similar messages:
2018/12/04 10:31:51.649633 WARNING data-sender Cannot connect to API: no link to API resource

Bests,
Istvan

2

I have an Nginx running infrot of the docker pmm server side. I have a feeling that this is some rewrite issue but I’m not sure what should I rewrite where…

3

I have the exact same problem. I am using Apache to terminate SSL and forward to the container locally.

4

Hello clouseau, gordan

Thanks for these posts. Can I just check a couple of things please?
[LIST]
[]It seems that you’re using a Proxy server in front of the PMM Server? Did we understand this correctly? If that’s the case, then you should know that’s an unsupported approach. Although the tech team will find it interesting I am VERY sure of that, it may take a while for them to get to address this. Having said that:
[]If you are using a Proxy, is that because of something you weren’t able to achieve with the Percona implementation for nginx? If that’s the case it would be good to be able to track that issue down. There is a sticky post in the PMM channel here that covers reporting of issues and bugs in case that’s helpful [url]https://www.percona.com/forums/questions-discussions/percona-monitoring-and-management/50690-pmm-troubleshooting-and-how-to-report-a-bug[/url]
[/LIST] In either case, the team would need a bit more to go on if they’re to provide some useful feedback and insights. So if you could let us know more about the scenario/environment you’re working in (and why if it’s to work around a PMM limitation!), and let us have copies of your nginx configs (any secure stuff edited out please), plus any error or access logs then I can find out if someone can take a look for you.

gordan - because you may well be encountering a different issue, it would be great if you could open up a separate thread? You should be able to subscribe to this one via your profile if you want to keep a watch on it. THANKS!

5

Hi gordan , clouseau - any detail to share on your proxy server setup? We’d like to see it working for you!

6

Hi,
Don’t know if this is the right post to do it but I am trying to setup the PMM-server behind a reverse proxy as well.

Let me explain the situation:
We currently have a pmm-server at our office. It is deployed on a citrix xenserver via OVA (version is 17.1 for servers as well as clients).
On this xenserver is a reverse proxy on another VM.
We have a firewall in our office that redirects incoming port 8443 → 443 internal. The reverse proxy (nginx) listens on port 443.

Then we have our production server with a pmm-client installed.

pmm-admin config is configured to use our office ip xxx.xxx.xxx.xxx:8443 as server with bind-address set to our production server’s ip.
whenever i run: “pmm-admin add mysql” the linux:metrics and mysql:metrics work as expected but mysql:queries fails with a 10s retry/timeout.
If I run this command with the --verbose flag then I can see that PUT’s to /qan-api/agents/{agentid}/cmd get a 404 not found response.
I see it try 10 times (1 every second for 10 seconds)

If I inspect the access.log from NGINX on the pmm-server I can see them incoming and have statuscode 404.

For testing I deployed a server similar to our production server on the xenserver.
It works fine to add a mysql monitor there when I choose the direct IP of the pmm-server as server.
If I choose the reverse proxy with LAN then I get the same error. Wich leads me to believe it is the fault of the reverse proxy.

I tried enabling websockets in the reverse proxy and other stuff like adding headers that I found on this forum but nothing seems to work.

I don’t know where to find more debug information. Sometimes I come across /var/log/qan-api.log on these forums but I don’t have that file on the pmm-server.

Any ideas where to look next?

7

The reason for the proxy is that several https sites in addition to PMM need to be in the same IP/port. So the only way to achieve it is to have a web server terminate TLS and then proxy the plain http to the PMM container IP.

8

I’d like to have an apache proxy in front of PMM as well. In my case, it’s so I can use the apache instance to force people through a (SAML) SSO auth. before they can view the PMM dashboards.

I think I may eventually try to solve this by having my DB hosts communicate to the docker container directly and just point end-users to the proxy (I can firewall things off so that people won’t be able to directly connect to the docker container’s instance). That’s a more complicated solution than just being able to run apache on the same host and have it proxy things to the docker container (which then only has to be available to the local host).

9

I’m back to troubleshooting this issue. On the Apache proxy side, there is nothing in the error log. In the access log:

10.1.2.3 - - [04/Oct/2020:19:04:47 +0100] “GET /qan-api/ping HTTP/1.1” 200 - “-” “Go-http-client/1.1”

10.1.2.3 - - [04/Oct/2020:19:04:47 +0100] “GET /v1/status/leader HTTP/1.1” 200 16 “-” “Go-http-client/1.1”

10.1.2.3 - - [04/Oct/2020:19:04:47 +0100] “GET /qan-api/ping HTTP/1.1” 401 172 “-” “Go-http-client/1.1”

10.1.2.3 - - [04/Oct/2020:19:04:47 +0100] “GET /managed/v1/version HTTP/1.1” 200 20 “-” “Go-http-client/1.1”

10.1.2.3 - - [04/Oct/2020:19:04:47 +0100] “GET /v1/catalog/node/data HTTP/1.1” 200 590 “-” “Go-http-client/1.1”

10.1.2.3 - - [04/Oct/2020:19:04:48 +0100] “GET /v1/catalog/node/data HTTP/1.1” 200 590 “-” “Go-http-client/1.1”

10.1.2.3 - - [04/Oct/2020:19:04:48 +0100] “GET /v1/catalog/node/data HTTP/1.1” 200 590 “-” “Go-http-client/1.1”

10.1.2.3 - - [04/Oct/2020:19:04:48 +0100] “GET /v1/catalog/service/mysql:queries?tag=alias_data HTTP/1.1” 200 2 “-” “Go-http-client/1.1”

10.1.2.3 - - [04/Oct/2020:19:04:48 +0100] “GET /v1/catalog/node/data HTTP/1.1” 200 590 “-” “Go-http-client/1.1”

10.1.2.3 - - [04/Oct/2020:19:04:48 +0100] “GET /qan-api/instances/dcd80f6d64e543097d6ee56e48d2acee HTTP/1.1” 200 237 “-” “Go-http-client/1.1”

10.1.2.3 - - [04/Oct/2020:19:04:48 +0100] “GET /qan-api/instances/?type=mysql&name=data&parent_uuid=a21bd52ba20d460a52e9b455f6cd4a2d HTTP/1.1” 200 301 “-” “Go-http-client/1.1”

10.1.2.3 - - [04/Oct/2020:19:04:48 +0100] “PUT /qan-api/instances/73d67e4241ba4c1e44c37ad06685b4c3 HTTP/1.1” 204 - “-” “Go-http-client/1.1”

10.1.2.3 - - [04/Oct/2020:19:04:48 +0100] “GET /qan-api/instances/73d67e4241ba4c1e44c37ad06685b4c3 HTTP/1.1” 200 301 “-” “Go-http-client/1.1”

10.1.2.3 - - [04/Oct/2020:19:04:49 +0100] “PUT /qan-api/agents/dcd80f6d64e543097d6ee56e48d2acee/cmd HTTP/1.1” 404 31 “-” “Go-http-client/1.1”

10.1.2.3 - - [04/Oct/2020:19:04:49 +0100] “GET /qan-api HTTP/1.1” 200 133 “-” “Go-http-client/1.1”

10.1.2.3 - - [04/Oct/2020:19:04:49 +0100] “GET /qan-api/agents/dcd80f6d64e543097d6ee56e48d2acee HTTP/1.1” 200 564 “-” “Go-http-client/1.1”

10.1.2.3 - - [04/Oct/2020:19:04:50 +0100] “PUT /qan-api/instances/73d67e4241ba4c1e44c37ad06685b4c3 HTTP/1.1” 204 - “-” “Go-http-client/1.1”

10.1.2.3 - - [04/Oct/2020:19:04:50 +0100] “GET /qan-api/agents/dcd80f6d64e543097d6ee56e48d2acee/log HTTP/1.1” 400 126 “-” “-”

10.1.2.3 - - [04/Oct/2020:19:04:50 +0100] “GET /qan-api/agents/dcd80f6d64e543097d6ee56e48d2acee/cmd HTTP/1.1” 404 220 “-” “-”

10.1.2.3 - - [04/Oct/2020:19:04:50 +0100] “PUT /qan-api/agents/dcd80f6d64e543097d6ee56e48d2acee/cmd HTTP/1.1” 404 31 “-” “Go-http-client/1.1”

10.1.2.3 - - [04/Oct/2020:19:04:51 +0100] “PUT /qan-api/agents/dcd80f6d64e543097d6ee56e48d2acee/cmd HTTP/1.1” 404 31 “-” “Go-http-client/1.1”

10.1.2.3 - - [04/Oct/2020:19:04:52 +0100] “PUT /qan-api/agents/dcd80f6d64e543097d6ee56e48d2acee/cmd HTTP/1.1” 404 31 “-” “Go-http-client/1.1”

10.1.2.3 - - [04/Oct/2020:19:04:53 +0100] “GET /qan-api/agents/dcd80f6d64e543097d6ee56e48d2acee/cmd HTTP/1.1” 404 220 “-” “-”

10.1.2.3 - - [04/Oct/2020:19:04:53 +0100] “GET /qan-api/agents/dcd80f6d64e543097d6ee56e48d2acee/log HTTP/1.1” 400 126 “-” “-”

10.1.2.3 - - [04/Oct/2020:19:04:53 +0100] “PUT /qan-api/agents/dcd80f6d64e543097d6ee56e48d2acee/cmd HTTP/1.1” 404 31 “-” “Go-http-client/1.1”

10.1.2.3 - - [04/Oct/2020:19:04:54 +0100] “PUT /qan-api/agents/dcd80f6d64e543097d6ee56e48d2acee/cmd HTTP/1.1” 404 31 “-” “Go-http-client/1.1”

10.1.2.3 - - [04/Oct/2020:19:04:55 +0100] “PUT /qan-api/agents/dcd80f6d64e543097d6ee56e48d2acee/cmd HTTP/1.1” 404 31 “-” “Go-http-client/1.1”

10.1.2.3 - - [04/Oct/2020:19:04:56 +0100] “PUT /qan-api/agents/dcd80f6d64e543097d6ee56e48d2acee/cmd HTTP/1.1” 404 31 “-” “Go-http-client/1.1”

10.1.2.3 - - [04/Oct/2020:19:04:57 +0100] “PUT /qan-api/agents/dcd80f6d64e543097d6ee56e48d2acee/cmd HTTP/1.1” 404 31 “-” “Go-http-client/1.1”

10.1.2.3 - - [04/Oct/2020:19:04:59 +0100] “PUT /qan-api/agents/dcd80f6d64e543097d6ee56e48d2acee/cmd HTTP/1.1” 404 31 “-” “Go-http-client/1.1”

10.1.2.3 - - [04/Oct/2020:19:05:00 +0100] “GET /qan-api/agents/dcd80f6d64e543097d6ee56e48d2acee/cmd HTTP/1.1” 404 220 “-” “-”

10.1.2.3 - - [04/Oct/2020:19:05:00 +0100] “GET /qan-api/agents/dcd80f6d64e543097d6ee56e48d2acee/log HTTP/1.1” 400 126 “-” “-”

Inside PMM Server container, in nginx log:

172.17.0.1 - pmm [04/Oct/2020:18:04:47 +0000] “GET /qan-api/ping HTTP/1.1” 200 0 “-” “Go-http-client/1.1” “10.1.2.3”

172.17.0.1 - pmm [04/Oct/2020:18:04:47 +0000] “GET /v1/status/leader HTTP/1.1” 200 16 “-” “Go-http-client/1.1” “10.1.2.3”

172.17.0.1 - - [04/Oct/2020:18:04:47 +0000] “GET /qan-api/ping HTTP/1.1” 401 172 “-” “Go-http-client/1.1” “10.1.2.3”

172.17.0.1 - pmm [04/Oct/2020:18:04:47 +0000] “GET /managed/v1/version HTTP/1.1” 200 20 “-” “Go-http-client/1.1” “10.1.2.3”

172.17.0.1 - pmm [04/Oct/2020:18:04:47 +0000] “GET /v1/catalog/node/data HTTP/1.1” 200 590 “-” “Go-http-client/1.1” “10.1.2.3”

172.17.0.1 - pmm [04/Oct/2020:18:04:48 +0000] “GET /v1/catalog/node/data HTTP/1.1” 200 590 “-” “Go-http-client/1.1” “10.1.2.3”

172.17.0.1 - pmm [04/Oct/2020:18:04:48 +0000] “GET /v1/catalog/node/data HTTP/1.1” 200 590 “-” “Go-http-client/1.1” “10.1.2.3”

172.17.0.1 - pmm [04/Oct/2020:18:04:48 +0000] “GET /v1/catalog/service/mysql:queries?tag=alias_data HTTP/1.1” 200 2 “-” “Go-http-client/1.1” “10.1.2.3”

172.17.0.1 - pmm [04/Oct/2020:18:04:48 +0000] “GET /v1/catalog/node/data HTTP/1.1” 200 590 “-” “Go-http-client/1.1” “10.1.2.3”

172.17.0.1 - pmm [04/Oct/2020:18:04:48 +0000] “GET /qan-api/instances/dcd80f6d64e543097d6ee56e48d2acee HTTP/1.1” 200 237 “-” “Go-http-client/1.1” “10.1.2.3”

172.17.0.1 - pmm [04/Oct/2020:18:04:48 +0000] “GET /qan-api/instances/?type=mysql&name=data&parent_uuid=a21bd52ba20d460a52e9b455f6cd4a2d HTTP/1.1” 200 301 “-” “Go-http-client/1.1” “10.1.2.3”

172.17.0.1 - pmm [04/Oct/2020:18:04:48 +0000] “PUT /qan-api/instances/73d67e4241ba4c1e44c37ad06685b4c3 HTTP/1.1” 204 0 “-” “Go-http-client/1.1” “10.1.2.3”

172.17.0.1 - pmm [04/Oct/2020:18:04:48 +0000] “GET /qan-api/instances/73d67e4241ba4c1e44c37ad06685b4c3 HTTP/1.1” 200 301 “-” “Go-http-client/1.1” “10.1.2.3”

172.17.0.1 - pmm [04/Oct/2020:18:04:49 +0000] “PUT /qan-api/agents/dcd80f6d64e543097d6ee56e48d2acee/cmd HTTP/1.1” 404 31 “-” “Go-http-client/1.1” “10.1.2.3”

172.17.0.1 - pmm [04/Oct/2020:18:04:49 +0000] “GET /qan-api HTTP/1.1” 200 133 “-” “Go-http-client/1.1” “10.1.2.3”

172.17.0.1 - pmm [04/Oct/2020:18:04:49 +0000] “GET /qan-api/agents/dcd80f6d64e543097d6ee56e48d2acee HTTP/1.1” 200 564 “-” “Go-http-client/1.1” “10.1.2.3”

172.17.0.1 - pmm [04/Oct/2020:18:04:50 +0000] “PUT /qan-api/instances/73d67e4241ba4c1e44c37ad06685b4c3 HTTP/1.1” 204 0 “-” “Go-http-client/1.1” “10.1.2.3”

172.17.0.1 - pmm [04/Oct/2020:18:04:50 +0000] “GET /qan-api/agents/dcd80f6d64e543097d6ee56e48d2acee/log HTTP/1.1” 400 126 “-” “-” "10.1.2.3"172.17.0.1 - pmm [04/Oct/2020:18:04:50 +0000] “GET /qan-api/agents/dcd80f6d64e543097d6ee56e48d2acee/cmd HTTP/1.1” 404 220 “-” “-” “10.1.2.3”

172.17.0.1 - pmm [04/Oct/2020:18:04:50 +0000] “PUT /qan-api/agents/dcd80f6d64e543097d6ee56e48d2acee/cmd HTTP/1.1” 404 31 “-” “Go-http-client/1.1” “10.1.2.3”

172.17.0.1 - pmm [04/Oct/2020:18:04:51 +0000] “PUT /qan-api/agents/dcd80f6d64e543097d6ee56e48d2acee/cmd HTTP/1.1” 404 31 “-” “Go-http-client/1.1” “10.1.2.3”

172.17.0.1 - pmm [04/Oct/2020:18:04:52 +0000] “PUT /qan-api/agents/dcd80f6d64e543097d6ee56e48d2acee/cmd HTTP/1.1” 404 31 “-” “Go-http-client/1.1” “10.1.2.3”

172.17.0.1 - pmm [04/Oct/2020:18:04:53 +0000] “GET /qan-api/agents/dcd80f6d64e543097d6ee56e48d2acee/cmd HTTP/1.1” 404 220 “-” “-” "10.1.2.3"172.17.0.1 - pmm [04/Oct/2020:18:04:53 +0000] “GET /qan-api/agents/dcd80f6d64e543097d6ee56e48d2acee/log HTTP/1.1” 400 126 “-” “-” “10.1.2.3”

172.17.0.1 - pmm [04/Oct/2020:18:04:53 +0000] “PUT /qan-api/agents/dcd80f6d64e543097d6ee56e48d2acee/cmd HTTP/1.1” 404 31 “-” “Go-http-client/1.1” “10.1.2.3”

172.17.0.1 - pmm [04/Oct/2020:18:04:54 +0000] “PUT /qan-api/agents/dcd80f6d64e543097d6ee56e48d2acee/cmd HTTP/1.1” 404 31 “-” “Go-http-client/1.1” “10.1.2.3”

172.17.0.1 - pmm [04/Oct/2020:18:04:55 +0000] “PUT /qan-api/agents/dcd80f6d64e543097d6ee56e48d2acee/cmd HTTP/1.1” 404 31 “-” “Go-http-client/1.1” “10.1.2.3”

172.17.0.1 - pmm [04/Oct/2020:18:04:56 +0000] “PUT /qan-api/agents/dcd80f6d64e543097d6ee56e48d2acee/cmd HTTP/1.1” 404 31 “-” “Go-http-client/1.1” “10.1.2.3”

172.17.0.1 - pmm [04/Oct/2020:18:04:57 +0000] “PUT /qan-api/agents/dcd80f6d64e543097d6ee56e48d2acee/cmd HTTP/1.1” 404 31 “-” “Go-http-client/1.1” “10.1.2.3”

172.17.0.1 - pmm [04/Oct/2020:18:04:59 +0000] “PUT /qan-api/agents/dcd80f6d64e543097d6ee56e48d2acee/cmd HTTP/1.1” 404 31 “-” “Go-http-client/1.1” “10.1.2.3”

172.17.0.1 - pmm [04/Oct/2020:18:05:00 +0000] “GET /qan-api/agents/dcd80f6d64e543097d6ee56e48d2acee/cmd HTTP/1.1” 404 220 “-” “-” "10.1.2.3"172.17.0.1 - pmm [04/Oct/2020:18:05:00 +0000] “GET /qan-api/agents/dcd80f6d64e543097d6ee56e48d2acee/log HTTP/1.1” 400 126 “-” “-” “10.1.2.3”

Looking through the commands above that work and the ones that don’t work:

https://pmm/qan-api/agents/dcd80f6d64e543097d6ee56e48d2acee

returns

{“UUID”:“dcd80f6d64e543097d6ee56e48d2acee”,“ParentUUID”:“a21bd52ba20d460a52e9b455f6cd4a2d”,“Hostname”:“data”,“Version”:“1.0.5”,“Created”:“2020-10-03T23:49:47Z”,“Deleted”:“0001-01-01T00:00:00Z”,“Links”:{“cmd”:“ws://pmm/qan-api/agents/dcd80f6d64e543097d6ee56e48d2acee/cmd”,“data”:“ws://pmm/qan-api/agents/dcd80f6d64e543097d6ee56e48d2acee/data”,“log”:“ws://pmm/qan-api/agents/dcd80f6d64e543097d6ee56e48d2acee/log”,“self”:"<a href=“http://pmm.shatteredsilicon.net/qan-api/agents/dcd80f6d64e543097d6ee56e48d2acee"}}">http://pmm/qan-api/agents/dcd80f6d64e543097d6ee56e48d2acee”}}

But:

https://pmm/qan-api/agents/dcd80f6d64e543097d6ee56e48d2acee/cmd

returns a 404:

Not Found

No matching route found: /agents/dcd80f6d64e543097d6ee56e48d2acee/cmd

Inside the container in the qan-api.log:

2020/10/04 18:04:47.766 127.0.0.1 200 127.365us GET /ping

2020/10/04 18:04:48.170 127.0.0.1 200 1.372578ms GET /instances/dcd80f6d64e543097d6ee56e48d2acee

2020/10/04 18:04:48.265 127.0.0.1 200 1.647799ms GET /instances/

2020/10/04 18:04:48.343 127.0.0.1 0 1.791326ms PUT /instances/73d67e4241ba4c1e44c37ad06685b4c3

2020/10/04 18:04:48.417 127.0.0.1 200 1.065216ms GET /instances/73d67e4241ba4c1e44c37ad06685b4c3

2020/10/04 18:04:49.341 127.0.0.1 404 1.404129ms PUT /agents/dcd80f6d64e543097d6ee56e48d2acee/cmd

2020/10/04 18:04:49.389 127.0.0.1 200 69.861us GET /

2020/10/04 18:04:49.403 127.0.0.1 200 1.147961ms GET /agents/dcd80f6d64e543097d6ee56e48d2acee

2020/10/04 18:04:50.285 127.0.0.1 0 1.968151ms PUT /instances/73d67e4241ba4c1e44c37ad06685b4c3

2020/10/04 18:04:50.350 127.0.0.1 400 1.715599ms GET /agents/dcd80f6d64e543097d6ee56e48d2acee/log

2020/10/04 18:04:50.355 127.0.0.1 404 83.411us GET /agents/dcd80f6d64e543097d6ee56e48d2acee/cmd

2020/10/04 18:04:50.417 127.0.0.1 404 1.612986ms PUT /agents/dcd80f6d64e543097d6ee56e48d2acee/cmd

2020/10/04 18:04:51.496 127.0.0.1 404 1.564227ms PUT /agents/dcd80f6d64e543097d6ee56e48d2acee/cmd

2020/10/04 18:04:52.569 127.0.0.1 404 1.643864ms PUT /agents/dcd80f6d64e543097d6ee56e48d2acee/cmd

2020/10/04 18:04:53.453 127.0.0.1 404 90.952us GET /agents/dcd80f6d64e543097d6ee56e48d2acee/cmd

2020/10/04 18:04:53.452 127.0.0.1 400 1.48291ms GET /agents/dcd80f6d64e543097d6ee56e48d2acee/log

2020/10/04 18:04:53.639 127.0.0.1 404 1.582219ms PUT /agents/dcd80f6d64e543097d6ee56e48d2acee/cmd

2020/10/04 18:04:54.712 127.0.0.1 404 1.766631ms PUT /agents/dcd80f6d64e543097d6ee56e48d2acee/cmd

2020/10/04 18:04:55.811 127.0.0.1 404 1.801726ms PUT /agents/dcd80f6d64e543097d6ee56e48d2acee/cmd

2020/10/04 18:04:56.884 127.0.0.1 404 1.794774ms PUT /agents/dcd80f6d64e543097d6ee56e48d2acee/cmd

2020/10/04 18:04:57.956 127.0.0.1 404 1.674902ms PUT /agents/dcd80f6d64e543097d6ee56e48d2acee/cmd

2020/10/04 18:04:59.053 127.0.0.1 404 1.489232ms PUT /agents/dcd80f6d64e543097d6ee56e48d2acee/cmd

2020/10/04 18:05:00.532 127.0.0.1 404 123.356us GET /agents/dcd80f6d64e543097d6ee56e48d2acee/cmd

So the error seems to genuinely be coming from the QAN API. I’m purely guessing that the agent creation fails here:

2020/10/04 18:04:49.341 127.0.0.1 404 1.404129ms PUT /agents/dcd80f6d64e543097d6ee56e48d2acee/cmd

but I am not sure how to troubleshoot it further. Any hints on where to look next would be gratefully received.

10

@Michael_Coburn the proxy server config in Apache is fairly trivial:

ProxyRequests      On

ProxyPreserveHost    On

    ProxyPass               /       <a href="http://localhost:10080/">http://localhost:10080/</a>

    ProxyPassReverse        /       <a href="http://localhost:10080/">http://localhost:10080/</a>

The docker container is started using

docker run -d \

-p 127.0.0.1:10080:80 \

–volumes-from pmm-data \

–name pmm \

–restart always \

percona/pmm-server:1

hence why the proxy directs traffic to localhost:10080.