pmm-admin config: Detected address '172.19.0.4' is not locally bound, then HTTP 400

[djvdorp]

I am trying to setup PMM Client on a dedicated server of mine, which only has a public IPv4 and public IPv6 IP.
However, for some reason beyond my understanding, PMM Client insists that there is a private address somewhere when I run this:

pmm-admin config --server pmm-server.example.com
Detected address ‘172.19.0.4’ is not locally bound.
This usually happens when client and server are on the different networks.

Use --bind-address flag to set locally bound address, usually a private one, while client address is public.
The bind address should correspond to the detected client address via NAT and you would need to configure port forwarding.

PMM server should be able to connect to the client address ‘172.19.0.4’ which should translate to a local bind address.
What ports to map you can find from “pmm-admin check-network” output once you add instances to the monitoring.

I found another forum post that suggested me to do this instead:

pmm-admin config --server PMM-SERVER-IP --bind-address PMM-CLIENT-IP --client-address PMM-CLIENT-IP

So when I ran that, with PMM-CLIENT-IP being the PMM Client host public IPv4 address, it continues:

OK, PMM server is alive.

PMM Server | pmm-server.example.com
Client Name | pmm-client.example.com
Client Address | PMM-CLIENT-IP

However, it does not actually function:

pmm-admin add mysql --create-user
[linux:metrics] Error adding linux metrics: Unexpected response code: 400 (Request decode failed: EOF)

Can you please explain me what I am doing wrong here, if any, and why would PMM Client insist so much on using a private IP address I don’t have configured?

[lorraine.pocklington]

Hello djvdorp can I just check, are you running PMM Client in a docker container?

[djvdorp]

Hello lorraine.pocklington – no, the PMM Client is directly installed on the dedicated server host system, which is Ubuntu 16.04 LTS.

[Michael_Coburn]

Hi djvdorp ,

That IP address of 172.19 looks very much the Docker internal subnet. From the host, if you run ip addr you should see a docker0 interface.

One way to avoid the Docker network is to launch your container using

docker run --net=host ...

• this way PMM Server will be directly attached to the host’s public IP address.

[djvdorp]

Hi Michael Coburn – I think your analysis is right and the 172.19 IP address is probably the Docker internal subnet. The reason this occurs is, that I am running the PMM Server container behind an nginx reverse proxy instance, https://github.com/jwilder/nginx-proxy to be more precise, so that I can use that + Lets Encrypt for SSL on my instance.

So while the regular command to run the instance is this:
docker run -d -p 80:80 --volumes-from pmm-data --name pmm-server --restart always percona/pmm-server:latest

I now run this instead:
docker run -d -e VIRTUAL_HOST=pmm-server.example.com -e LETSENCRYPT_HOST=pmm-server.example.com -e LETSENCRYPT_EMAIL=me@example.com --volumes-from pmm-data --network=“services” --name pmm-server --restart always percona/pmm-server:latest

Then the reverse proxy nginx instance will pick up that pmm-server exposes port 80 and puts itself in front of it so that it can HTTP 80 and HTTPS 443 for it.

However, I genuinely believe that the fact that there seems to be no way to override the ‘172.19.0.4’ that pmm-admin detects is a bug; there should be a -f/–force flag to not detect the ‘172.19.0.4’ or at least ignore it in the case that I just described above; the way I worked around it for now is to stop my reverse proxy setup, run it like documented (so command 1 seen above), just to add the server to pmm-admin via ‘pmm-admin config --server PMM-SERVER-IP’ (which works fine) and then run it like I want it again, including the reverse proxy setup with nginx for HTTP and HTTPS (so command 2 seen above). This is a sub-ideal solution to this problem however, hence my trying to improve this situation.

–note added by Percona Community Manager, going forward using pmm-server:n rather than pmm-server:latest is best practice, our advice changed since this post was written. In this case pmm-server:1 would have been appropriate for these code examples.

[lorraine.pocklington]

Thanks for spending the time on this feedback, and also for documenting the solution as it might help other users in the same scenario - we appreciate your input.

There’s a documented approach to SSL that you’ve probably seen, but just in case other users come across this post I should share the link

https://www.percona.com/doc/percona-monitoring-and-management/security.html#enabling-ssl-encryption